Gaze-based user interactions

ABSTRACT

In an exemplary process for interacting with user interface objects using an eye gaze, an affordance associated with a first object is displayed. A gaze direction or a gaze depth is determined. While the gaze direction or the gaze depth is determined to correspond to a gaze at the affordance, a first input representing user instruction to take action on the affordance is received, and the affordance is selected responsive to receiving the first input.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/US2018/053428, entitled “Gaze-Based User Interactions,” filed Sep.28, 2018, which claims priority to: U.S. Patent Application Ser. No.62/566,073, entitled “Accessing Functions of External Devices UsingReality Interfaces,” filed Sep. 29, 2017; U.S. Patent Application Ser.No. 62/566,080, entitled “Controlling External Devices Using RealityInterfaces,” filed Sep. 29, 2017; U.S. Patent Application Ser. No.62/566,206, entitled “Gaze-based User Interactions,” filed Sep. 29,2017; and U.S. Patent Application Ser. No. 62/734,678, entitled“Gaze-based User Interactions,” filed Sep. 21, 2018, which are herebyincorporated by reference in their entirety for all purposes.

BACKGROUND 1. Field

The present disclosure relates generally to user interfaces forinteracting with an electronic device, and more specifically tointeracting with an electronic device using an eye gaze.

2. Description of Related Art

Conventional electronic devices use input mechanisms, such as keyboards,buttons, joysticks, and touch-screens, to receive inputs from a user.Some conventional devices also include a screen that displays contentresponsive to a user's input. Such input mechanisms and displays providean interface for the user to interact with an electronic device.

BRIEF SUMMARY

The present disclosure describes techniques for interacting with anelectronic device using an eye gaze. According to some embodiments, auser uses his or her eyes to interact with user interface objectsdisplayed on the electronic device. The techniques provide a morenatural and efficient interface by, in some exemplary embodiments,allowing a user to operate the device using primarily eye gazes and eyegestures (e.g., eye movement, blinks, and stares). Techniques are alsodescribed for using eye gaze to quickly designate an initial position(e.g., for selecting or placing an object) and then moving thedesignated position without using eye gaze, as precisely locating thedesignated position can be difficult using eye gaze due to uncertaintyand instability of the position of a user's eye gaze. The techniques canbe applied to conventional user interfaces on devices such as desktopcomputers, laptops, tablets, and smartphones. The techniques are alsoadvantageous for computer-generated reality (including virtual realityand mixed reality) devices and applications, as described in greaterdetail below.

According to some embodiments, an affordance associated with a firstdisplayed object is displayed and a gaze direction or a gaze depth isdetermined. A determination is made whether the gaze direction or thegaze depth corresponds to a gaze at the affordance. A first inputrepresenting an instruction to take action on the affordance is receivedwhile the gaze direction or the gaze depth is determined to correspondto a gaze at the affordance, and the affordance is selected responsiveto receiving the first input.

According to some embodiments, a first affordance and a secondaffordance are concurrently displayed and a first gaze direction or afirst gaze depth of one or more eyes is determined. A determination ismade whether the first gaze direction or the first gaze depthcorresponds to a gaze at both the first affordance and the secondaffordance. In response to determining that the first gaze direction orthe first gaze depth corresponds to a gaze at both the first affordanceand the second affordance, the first affordance and the secondaffordance are enlarged.

According to some embodiments, an electronic device adapted to display afield of view of a three-dimensional computer generated realityenvironment and the field of view is rendered from a viewingperspective. A first object is displayed concurrently with a secondobject, where the first object is presented closer than the secondobject from the viewing position. A gaze position is determined. Inaccordance with a determination that the gaze position corresponds to agaze at the first object, the display of the second object is visuallyaltered. In accordance with a determination that the gaze positioncorresponds to a gaze at the second object, the display of the firstobject is visually altered.

According to some embodiments, a first user input is received at a firsttime. In response to receiving the first user input, a selection pointis designated at a first position corresponding to a gaze position atthe first time. While maintaining designation of the selection point asecond user input is received. In response to receiving the second userinput, the selection point is moved to a second position different thanthe first position, where moving the selection point to the secondposition is not based on the gaze position. While the selection point isat the second position, a third user input is received. In response toreceiving the third user input, the selection point is confirmed at thesecond position.

According to some embodiments, a first user input is received at a firsttime. In response to receiving the first user input, a first object of aplurality of objects corresponding to a gaze position at the first timeis designated. While maintaining designation of the first object, asecond user input is received. In response to receiving the second userinput, designation of the first object is ceased and a second object ofthe plurality of objects is designated, where designating the secondobject is not based on the gaze position. While maintaining designationof the second object, a third user input is received. In response toreceiving the third user input, the second object is selected.

According to some embodiments, an object is selected. While maintainingselection of the object, a first user input is received at a first time.In response to receiving the first user input, a placement point isdesignated at a first position based on a gaze position at the firsttime, where the first position corresponds to the gaze position at thefirst time. While maintaining designation of the placement point, asecond user input is received. In response to receiving the second userinput, the placement point is moved to a second position different thanthe first position, where moving the placement point to the secondposition is not based on the gaze position. A third user input isreceived, and in response to receiving the third user input, theselected object is placed at the second position.

DESCRIPTION OF THE FIGURES

For a better understanding of the various described embodiments,reference should be made to the Detailed Description below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIGS. 1A and 1B depict exemplary systems for use in variouscomputer-generated reality technologies, including virtual reality andmixed reality.

FIGS. 1C-1E illustrate embodiments of the system in the form of mobiledevices.

FIGS. 1F-1H illustrate embodiments of the system in the form of headmounted display (HMD) devices.

FIG. 1I illustrates an embodiment of the system in the form of a head-updisplay (HUD) device.

FIG. 2 illustrates a user viewing an object, according to variousembodiments.

FIG. 3 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 4 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 5 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 6 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 7 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 8 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 9 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 10 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 11 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 12 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 13 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 14 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 15 illustrates a user interface for interacting with an electronicdevice using an eye gaze, according to various embodiments.

FIG. 16 depicts a flow chart of an exemplary process for interactingwith an electronic device using an eye gaze, according to variousembodiments.

FIG. 17 depicts a flow chart of an exemplary process for interactingwith an electronic device using an eye gaze, according to variousembodiments.

FIG. 18 depicts a flow chart of an exemplary process for interactingwith an electronic device using an eye gaze, according to variousembodiments.

FIGS. 19A-19Y illustrate a user interface for interacting with anelectronic device using an eye gaze, according to various embodiments.

FIG. 20 depicts a flow chart of an exemplary process for interactingwith an electronic device using an eye gaze, according to variousembodiments.

FIG. 21 depicts a flow chart of an exemplary process for interactingwith an electronic device using an eye gaze, according to variousembodiments.

FIG. 22 depicts a flow chart of an exemplary process for interactingwith an electronic device using an eye gaze, according to variousembodiments.

DETAILED DESCRIPTION

The following description sets forth exemplary methods, parameters, andthe like. It should be recognized, however, that such description is notintended as a limitation on the scope of the present disclosure but isinstead provided as a description of exemplary embodiments.

Various embodiments of electronic systems and techniques for using suchsystems in relation to various computer-generated reality technologies,including virtual reality and mixed reality (which incorporates sensoryinputs from a physical environment), are described.

A physical environment (or real environment) refers to a physical worldthat people can sense and/or interact with without aid of electronicsystems. Physical environments, such as a physical park, includephysical articles (or physical objects or real objects), such asphysical trees, physical buildings, and physical people. People candirectly sense and/or interact with the physical environment, such asthrough sight, touch, hearing, taste, and smell.

In contrast, a computer-generated reality (CGR) environment refers to awholly or partially simulated environment that people sense and/orinteract with via an electronic system. In CGR, a subset of a person'sphysical motions, or representations thereof, are tracked, and, inresponse, one or more characteristics of one or more virtual objectssimulated in the CGR environment are adjusted in a manner that comportswith at least one law of physics. For example, a CGR system may detect aperson's head turning and, in response, adjust graphical content and anacoustic field presented to the person in a manner similar to how suchviews and sounds would change in a physical environment. In somesituations (e.g., for accessibility reasons), adjustments tocharacteristic(s) of virtual object(s) in a CGR environment may be madein response to representations of physical motions (e.g., vocalcommands).

A person may sense and/or interact with a CGR object using any one oftheir senses, including sight, sound, touch, taste, and smell. Forexample, a person may sense and/or interact with audio objects thatcreate a 3D or spatial audio environment that provides the perception ofpoint audio sources in 3D space. In another example, audio objects mayenable audio transparency, which selectively incorporates ambient soundsfrom the physical environment with or without computer-generated audio.In some CGR environments, a person may sense and/or interact only withaudio objects.

Examples of CGR include virtual reality and mixed reality.

A virtual reality (VR) environment (or virtual environment) refers to asimulated environment that is designed to be based entirely oncomputer-generated sensory inputs for one or more senses. A VRenvironment comprises a plurality of virtual objects with which a personmay sense and/or interact. For example, computer-generated imagery oftrees, buildings, and avatars representing people are examples ofvirtual objects. A person may sense and/or interact with virtual objectsin the VR environment through a simulation of the person's presencewithin the computer-generated environment, and/or through a simulationof a subset of the person's physical movements within thecomputer-generated environment.

In contrast to a VR environment, which is designed to be based entirelyon computer-generated sensory inputs, a mixed reality (MR) environmentrefers to a simulated environment that is designed to incorporatesensory inputs from the physical environment, or a representationthereof, in addition to including computer-generated sensory inputs(e.g., virtual objects). On a virtuality continuum, a mixed realityenvironment is anywhere between, but not including, a wholly physicalenvironment at one end and virtual reality environment at the other end.

In some MR environments, computer-generated sensory inputs may respondto changes in sensory inputs from the physical environment. Also, someelectronic systems for presenting an MR environment may track locationand/or orientation with respect to the physical environment to enablevirtual objects to interact with real objects (that is, physicalarticles from the physical environment or representations thereof). Forexample, a system may account for movements so that a virtual treeappears stationary with respect to the physical ground.

Examples of mixed realities include augmented reality and augmentedvirtuality.

An augmented reality (AR) environment refers to a simulated environmentin which one or more virtual objects are superimposed over a physicalenvironment, or a representation thereof. For example, an electronicsystem for presenting an AR environment may have a transparent ortranslucent display through which a person may directly view thephysical environment. The system may be configured to present virtualobjects on the transparent or translucent display, so that a person,using the system, perceives the virtual objects superimposed over thephysical environment. Alternatively, a system may have an opaque displayand one or more imaging sensors that capture images or video of thephysical environment, which are representations of the physicalenvironment. The system composites the images or video with virtualobjects, and presents the composition on the opaque display. A person,using the system, indirectly views the physical environment by way ofthe images or video of the physical environment, and perceives thevirtual objects superimposed over the physical environment. As usedherein, a video of the physical environment shown on an opaque displayis called “pass-through video,” meaning a system uses one or more imagesensor(s) to capture images of the physical environment, and uses thoseimages in presenting the AR environment on the opaque display. Furtheralternatively, a system may have a projection system that projectsvirtual objects into the physical environment, for example, as ahologram or on a physical surface, so that a person, using the system,perceives the virtual objects superimposed over the physicalenvironment.

An augmented reality environment also refers to a simulated environmentin which a representation of a physical environment is transformed bycomputer-generated sensory information. For example, in providingpass-through video, a system may transform one or more sensor images toimpose a select perspective (e.g., viewpoint) different than theperspective captured by the imaging sensors. As another example, arepresentation of a physical environment may be transformed bygraphically modifying (e.g., enlarging) portions thereof, such that themodified portion may be representative but not photorealistic versionsof the originally captured images. As a further example, arepresentation of a physical environment may be transformed bygraphically eliminating or obfuscating portions thereof.

An augmented virtuality (AV) environment refers to a simulatedenvironment in which a virtual or computer generated environmentincorporates one or more sensory inputs from the physical environment.The sensory inputs may be representations of one or more characteristicsof the physical environment. For example, an AV park may have virtualtrees and virtual buildings, but people with faces photorealisticallyreproduced from images taken of physical people. As another example, avirtual object may adopt a shape or color of a physical article imagedby one or more imaging sensors. As a further example, a virtual objectmay adopt shadows consistent with the position of the sun in thephysical environment.

There are many different types of electronic systems that enable aperson to sense and/or interact with various CGR environments. Examplesinclude head mounted systems, projection-based systems, heads-updisplays (HUDs), vehicle windshields having integrated displaycapability, windows having integrated display capability, displaysformed as lenses designed to be placed on a person's eyes (e.g., similarto contact lenses), headphones/earphones, speaker arrays, input systems(e.g., wearable or handheld controllers with or without hapticfeedback), smartphones, tablets, and desktop/laptop computers. A headmounted system may have one or more speaker(s) and an integrated opaquedisplay. Alternatively, a head mounted system may be configured toaccept an external opaque display (e.g., a smartphone). The head mountedsystem may incorporate one or more imaging sensors to capture images orvideo of the physical environment, and/or one or more microphones tocapture audio of the physical environment. Rather than an opaquedisplay, a head mounted system may have a transparent or translucentdisplay. The transparent or translucent display may have a mediumthrough which light representative of images is directed to a person'seyes. The display may utilize digital light projection, OLEDs, LEDs,uLEDs, liquid crystal on silicon, laser scanning light source, or anycombination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In one embodiment, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface.

FIGS. 1A and 1B depict exemplary system 100 for use in variouscomputer-generated reality technologies, including virtual reality andmixed reality.

In some embodiments, as illustrated in FIG. 1A, system 100 includesdevice 100 a. Device 100 a includes various components, such asprocessor(s) 102, RF circuitry(ies) 104, memory(ies) 106, imagesensor(s) 108, orientation sensor(s) 110, microphone(s) 112, locationsensor(s) 116, speaker(s) 118, display(s) 120, and touch-sensitivesurface(s) 122. These components optionally communicate overcommunication bus(es) 150 of device 100 a.

In some embodiments, elements of system 100 are implemented in a basestation device (e.g., a computing device, such as a remote server,mobile device, or laptop) and other elements of the system 100 areimplemented in a head-mounted display (HMD) device designed to be wornby the user, where the HMD device is in communication with the basestation device. In some embodiments, device 100 a is implemented in abase station device or a HMD device.

As illustrated in FIG. 1B, in some embodiments, system 100 includes two(or more) devices in communication, such as through a wired connectionor a wireless connection. First device 100 b (e.g., a base stationdevice) includes processor(s) 102, RF circuitry(ies) 104, andmemory(ies) 106. These components optionally communicate overcommunication bus(es) 150 of device 100 b. Second device 100 c (e.g., ahead-mounted device) includes various components, such as processor(s)102, RF circuitry(ies) 104, memory(ies) 106, image sensor(s) 108,orientation sensor(s) 110, microphone(s) 112, location sensor(s) 116,speaker(s) 118, display(s) 120, and touch-sensitive surface(s) 122.These components optionally communicate over communication bus(es) 150of device 100 c.

In some embodiments, system 100 is a mobile device, such as in theembodiments described with respect to device 100 a in FIGS. 1C-1E. Insome embodiments, system 100 is a head-mounted display (HMD) device,such as in the embodiments described with respect to device 100 a inFIGS. 1F-1H. In some embodiments, system 100 is a wearable HUD device,such as in the embodiments described with respect to device 100 a inFIG. 1I.

System 100 includes processor(s) 102 and memory(ies) 106. Processor(s)102 include one or more general processors, one or more graphicsprocessors, and/or one or more digital signal processors. In someembodiments, memory(ies) 106 are one or more non-transitorycomputer-readable storage mediums (e.g., flash memory, random accessmemory) that store computer-readable instructions configured to beexecuted by processor(s) 102 to perform the techniques described below.

System 100 includes RF circuitry(ies) 104. RF circuitry(ies) 104optionally include circuitry for communicating with electronic devices,networks, such as the Internet, intranets, and/or a wireless network,such as cellular networks and wireless local area networks (LANs). RFcircuitry(ies) 104 optionally includes circuitry for communicating usingnear-field communication and/or short-range communication, such asBluetooth®.

System 100 includes display(s) 120. In some embodiments, display(s) 120include a first display (e.g., a left eye display panel) and a seconddisplay (e.g., a right eye display panel), each display for displayingimages to a respective eye of the user. Corresponding images aresimultaneously displayed on the first display and the second display.Optionally, the corresponding images include the same virtual objectsand/or representations of the same physical objects from differentviewpoints, resulting in a parallax effect that provides a user with theillusion of depth of the objects on the displays. In some embodiments,display(s) 120 include a single display. Corresponding images aresimultaneously displayed on a first area and a second area of the singledisplay for each eye of the user. Optionally, the corresponding imagesinclude the same virtual objects and/or representations of the samephysical objects from different viewpoints, resulting in a parallaxeffect that provides a user with the illusion of depth of the objects onthe single display.

In some embodiments, system 100 includes touch-sensitive surface(s) 122for receiving user inputs, such as tap inputs and swipe inputs. In someembodiments, display(s) 120 and touch-sensitive surface(s) 122 formtouch-sensitive display(s).

System 100 includes image sensor(s) 108. Image sensors(s) 108 optionallyinclude one or more visible light image sensor, such as charged coupleddevice (CCD) sensors, and/or complementary metal-oxide-semiconductor(CMOS) sensors operable to obtain images of physical objects from thereal environment. Image sensor(s) also optionally include one or moreinfrared (IR) sensor(s), such as a passive IR sensor or an active IRsensor, for detecting infrared light from the real environment. Forexample, an active IR sensor includes an IR emitter, such as an IR dotemitter, for emitting infrared light into the real environment. Imagesensor(s) 108 also optionally include one or more event camera(s)configured to capture movement of physical objects in the realenvironment. Image sensor(s) 108 also optionally include one or moredepth sensor(s) configured to detect the distance of physical objectsfrom system 100. In some embodiments, system 100 uses CCD sensors, eventcameras, and depth sensors in combination to detect the physicalenvironment around system 100. In some embodiments, image sensor(s) 108include a first image sensor and a second image sensor. The first imagesensor and the second image sensor are optionally configured to captureimages of physical objects in the real environment from two distinctperspectives. In some embodiments, system 100 uses image sensor(s) 108to receive user inputs, such as hand gestures. In some embodiments,system 100 uses image sensor(s) 108 to detect the position andorientation of system 100 and/or display(s) 120 in the real environment.For example, system 100 uses image sensor(s) 108 to track the positionand orientation of display(s) 120 relative to one or more fixed objectsin the real environment.

In some embodiments, system 100 includes microphones(s) 112. System 100uses microphone(s) 112 to detect sound from the user and/or the realenvironment of the user. In some embodiments, microphone(s) 112 includesan array of microphones (including a plurality of microphones) thatoptionally operate in tandem, such as to identify ambient noise or tolocate the source of sound in space of the real environment.

System 100 includes orientation sensor(s) 110 for detecting orientationand/or movement of system 100 and/or display(s) 120. For example, system100 uses orientation sensor(s) 110 to track changes in the positionand/or orientation of system 100 and/or display(s) 120, such as withrespect to physical objects in the real environment. Orientationsensor(s) 110 optionally include one or more gyroscopes and/or one ormore accelerometers.

FIGS. 1C-1E illustrate embodiments of system 100 in the form of device100 a. In FIGS. 1C-1E, device 100 a is a mobile device, such as acellular phone. FIG. 1C illustrates device 100 a carrying out a virtualreality technique. Device 100 a is displaying, on display 120, a virtualenvironment 160 that includes virtual objects, such as sun 160 a, birds160 b, and beach 160 c. Both the displayed virtual environment 160 andvirtual objects (e.g., 160 a, 160 b, 160 c) of the virtual environment160 are computer-generated imagery. Note that the virtual realityenvironment depicted in FIG. 1C does not include representations ofphysical objects from the real environment 180, such as physical person180 a and physical tree 180 b, even though these elements of realenvironment 180 are within the field of view of image sensor(s) 108 ofdevice 100 a.

FIG. 1D illustrates device 100 a carrying out a mixed reality technique,and in particular an augmented reality technique, using pass-throughvideo. Device 100 a is displaying, on display 120, a representation 170of the real environment 180 with virtual objects. The representation 170of the real environment 180 includes representation 170 a of person 180a and representation 170 b of tree 180 b. For example, the device usesimage sensor(s) 108 to capture images of the real environment 180 thatare passed through for display on display 120. Device 100 a overlays hat160 d, which is a virtual object generated by device 100 a, on the headof the representation 170 a of person 180 a. Device 100 a tracks thelocation and/or orientation of physical objects with respect to theposition and/or orientation of device 100 a to enable virtual objects tointeract with physical objects from the real environment in theaugmented reality environment. In this embodiment, device 100 a accountsfor movements of device 100 a and person 180 a to display hat 160 d asbeing on the head of the representation 170 a of person 180 a, even asdevice 100 a and person 180 a move relative to one another.

FIG. 1E illustrates device 100 a carrying out a mixed reality technique,and in particular an augmented virtuality technique. Device 100 a isdisplaying, on display 120, a virtual environment 160 withrepresentations of physical objects. The virtual environment 160includes virtual objects (e.g., sun 160 a, birds 160 b) andrepresentation 170 a of person 180 a. For example, device 100 a usesimage sensor(s) 108 to capture images of person 180 a in realenvironment 180. Device 100 a places representation 170 a of person 180a in virtual environment 160 for display on display 120. Device 100 aoptionally tracks the location and/or orientation of physical objectswith respect to the position and/or orientation of device 100 a toenable virtual objects to interact with physical objects from realenvironment 180. In this embodiment, device 100 a accounts for movementsof device 100 a and person 180 a to display hat 160 d as being on thehead of representation 170 a of person 180 a. Notably, in thisembodiment, device 100 a does not display a representation of tree 180 beven though tree 180 b is also within the field of view of the imagesensor(s) of device 100 a, in carrying out the mixed reality technique.

FIGS. 1F-1H illustrate embodiments of system 100 in the form of device100 a. In FIGS. 1F-1H, device 100 a is a HMD device configured to beworn on the head of a user, with each eye of the user viewing arespective display 120 a and 120 b. FIG. 1F illustrates device 100 acarrying out a virtual reality technique. Device 100 a is displaying, ondisplays 120 a and 120 b, a virtual environment 160 that includesvirtual objects, such as sun 160 a, birds 160 b, and beach 160 c. Thedisplayed virtual environment 160 and virtual objects (e.g., 160 a, 160b, 160 c) are computer-generated imagery. In this embodiment, device 100a simultaneously displays corresponding images on display 120 a anddisplay 120 b. The corresponding images include the same virtualenvironment 160 and virtual objects (e.g., 160 a, 160 b, 160 c) fromdifferent viewpoints, resulting in a parallax effect that provides auser with the illusion of depth of the objects on the displays. Notethat the virtual reality environment depicted in FIG. 1F does notinclude representations of physical objects from the real environment,such as person 180 a and tree 180 b even though person 180 a and tree180 b are within the field of view of the image sensor(s) of device 100a, in carrying out the virtual reality technique.

FIG. 1G illustrates device 100 a carrying out an augmented realitytechnique using pass-through video. Device 100 a is displaying, ondisplays 120 a and 120 b, a representation 170 of real environment 180with virtual objects. The representation 170 of real environment 180includes representation 170 a of person 180 a and representation 170 bof tree 180 b. For example, device 100 a uses image sensor(s) 108 tocapture images of the real environment 180 that are passed through fordisplay on displays 120 a and 120 b. Device 100 a is overlaying acomputer-generated hat 160 d (a virtual object) on the head ofrepresentation 170 a of person 180 a for display on each of displays 120a and 120 b. Device 100 a tracks the location and/or orientation ofphysical objects with respect to the position and/or orientation ofdevice 100 a to enable virtual objects to interact with physical objectsfrom real environment 180. In this embodiment, device 100 a accounts formovements of device 100 a and person 180 a to display hat 160 d as beingon the head of representation 170 a of person 180 a.

FIG. 1H illustrates device 100 a carrying out a mixed reality technique,and in particular an augmented virtuality technique, using pass-throughvideo. Device 100 a is displaying, on displays 120 a and 120 b, avirtual environment 160 with representations of physical objects. Thevirtual environment 160 includes virtual objects (e.g., sun 160 a, birds160 b) and representation 170 a of person 180 a. For example, device 100a uses image sensor(s) 108 to capture images of person 180 a. Device 100a places the representation 170 a of the person 180 a in the virtualenvironment for display on displays 120 a and 120 b. Device 100 aoptionally tracks the location and/or orientation of physical objectswith respect to the position and/or orientation of device 100 a toenable virtual objects to interact with physical objects from realenvironment 180. In this embodiment, device 100 a accounts for movementsof device 100 a and person 180 a to display hat 160 d as being on thehead of the representation 170 a of person 180 a. Notably, in thisembodiment, device 100 a does not display a representation of tree 180 beven though tree 180 b is also within the field of view of the imagesensor(s) 108 of device 100 a, in carrying out the mixed realitytechnique.

FIG. 1I illustrates an embodiment of system 100 in the form of device100 a. In FIG. 1I, device 100 a is a HUD device (e.g., a glasses device)configured to be worn on the head of a user, with each eye of the userviewing a respective heads-up display 120 c and 120 d. FIG. 1Iillustrates device 100 a carrying out an augmented reality techniqueusing heads-up displays 120 c and 120 d. The heads-up displays 120 c and120 d are (at least partially) transparent displays, thus allowing theuser to view the real environment 180 in combination with heads-updisplays 120 c and 120 d. Device 100 a is displaying, on each ofheads-up displays 120 c and 120 d, a virtual hat 160 d (a virtualobject). The device 100 a tracks the location and/or orientation ofphysical objects in the real environment with respect to the positionand/or orientation of device 100 a and with respect to the position ofthe user's eyes to enable virtual objects to interact with physicalobjects from real environment 180. In this embodiment, device 100 aaccounts for movements of device 100 a, movements of the user's eyeswith respect to device 100 a, and movements of person 180 a to displayhat 160 d at locations on displays 120 c and 120 d such that it appearsto the user that the hat 160 d is on the head of person 180 a.

With reference now to FIGS. 2-15, exemplary techniques for interactingwith an electronic device using an eye gaze are described.

FIG. 2 depicts a top view of user 200 whose gaze is focused on object210. The user's gaze is defined by the visual axes of each of the user'seyes. The direction of the visual axes define the user's gaze direction,and the distance at which the axes converge defines the gaze depth. Thegaze direction can also be referred to as the gaze vector orline-of-sight. In FIG. 2, the gaze direction is in the direction ofobject 210 and the gaze depth is the distance D relative to the user.

In some embodiments, the center of the user's cornea, the center of theuser's pupil, and/or the center of rotation of the user's eyeball aredetermined to determine the position of the visual axis of the user'seye, and can therefore be used to determine the user's gaze directionand/or gaze depth. In some embodiments, gaze depth is determined basedon a point of convergence of the visual axes of the user's eyes (or alocation of minimum distance between the visual axes of the user's eyes)or some other measurement of the focus of a user's eye(s). Optionally,the gaze depth is used to estimate the distance at which the user's eyesare focused.

In FIG. 2, rays 201A and 201B are cast along the visual axes of the leftand right eyes of user 200, respectively, and are optionally used todetermine the user's gaze direction and/or gaze depth in what isreferred to as ray casting. FIG. 2 also depicts cones 202A and 202Bhaving angular extents 203A and 203B, respectively. Cones 202A and 202Bare also cast along the visual axes of the left and right eyes of user200, respectively, and are optionally used to determine the user's gazedirection and/or gaze depth in what is referred to as cone casting. Gazedirection and gaze depth often cannot be determined with absoluteaccuracy or precision due to factors such as eye motion, sensor motion,sampling frequency, sensor latency, sensor resolution, sensormisalignment, etc. Accordingly, in some embodiments, an angularresolution or (estimated) angular error is associated with gazedirection. In some embodiments, a depth resolution is associated withgaze depth. Optionally, the angular extent of the cone(s) (e.g., angularextents 203A and 203B of cones 202A and 202B, respectively) representsthe angular resolution of the user's gaze direction.

FIG. 3 depicts electronic device 300 with display 302. Electronic device300 displays virtual environment 304, which includes virtual object 306.In some embodiments, environment 304 is a CGR environment (e.g., a VR orMR environment). In the illustrated embodiment, object 306 is anaffordance with which user 200 can interact using a gaze. In someembodiments, affordance 306 is associated with a physical object (e.g.,an appliance or other device that can be controlled via interaction withaffordance 306). FIG. 3 also depicts a view from above user 200 thatshows the gaze direction of user 200. The visual axes of each of theuser's eyes are extrapolated onto a plane of the displayedrepresentation of virtual environment 304, which corresponds to theplane of display 302 of device 300. Spot 308 represents the gazedirection of user 200 on display 302.

As shown in FIG. 3, the gaze direction of user 200 corresponds to thedirection of affordance 306. The term “affordance” refers to a graphicaluser interface object with which a user can interact. Examples ofaffordances include user-interactive images (e.g., icons), buttons, andtext (e.g., hyperlinks). Electronic device 300 is configured todetermine the gaze direction of user 200. Device 300 captures data froma sensor directed toward the user and determines the gaze directionbased on the data captured from the sensor. In some embodiments in whicha three-dimensional representation of a scene is presented, such as theembodiment described below with respect to FIGS. 9-12, device 300 also(or alternatively) determines a gaze depth and whether the gaze depthcorresponds to affordance 306. Optionally, determining whether the gazedepth corresponds to the depth of the affordance is based at least inpart on the depth resolution of the gaze depth.

In the illustrated embodiment, device 300 includes image sensor 310,which is directed toward user 200 and captures image data of the eyes ofuser 200. In some embodiments, device 300 includes an event camera thatdetects event data from a user (e.g., the user's eyes) based on changesin detected light intensity over time and uses the event data todetermine gaze direction and/or gaze depth. Optionally, device 300 usesboth image data and event data (e.g., from an image sensor and aseparate event camera or a sensor configured to capture image data andevent data) to determine gaze direction and/or gaze depth. Optionally,device 300 uses ray casting and/or cone casting to determine the gazedirection and/or gaze depth.

Based on the gaze direction, device 300 determines that the gazedirection corresponds to affordance 306, since the gaze direction is inthe same direction as affordance 306 (e.g., rays or cones cast from theeyes of user 200 at least partially intersect affordance 306 or arewithin a margin of error of affordance 306). Optionally, determiningthat the gaze direction corresponds to affordance 306 is based at leastin part on an angular resolution of the gaze direction. In someembodiments in which a three-dimensional representation of a scene ispresented, device 300 also (or alternatively) determines whether thegaze depth corresponds to a depth of affordance 306. Optionally,determining whether the gaze depth corresponds to the depth of theaffordance is based at least in part on a depth resolution of the gazedepth. Optionally, affordance 306 is also located at the gaze depth (orwithin a depth range that is based on the depth resolution of the gazedepth).

In some embodiments, a gaze direction and/or gaze depth is determined tocontinue to correspond to a gaze at an affordance even after the gazedirection and/or gaze depth no longer overlaps with the affordance(e.g., once the gaze direction and/or gaze depth is initially determinedto correspond to a gaze at the affordance, the gaze direction and/orgaze depth is considered to correspond to a gaze at the affordance forat least a predetermined amount of time or for a predetermined amount oftime after the user looks away from the affordance).

While the gaze direction is determined to correspond to a gaze ataffordance 306, device 300 receives an input (referred to as a“confirming action”) that represents an instruction to take action onthe affordance corresponding to the first object. For example, theconfirming action is received while user 200 is determined to be lookingat affordance 306.

In response to receiving the confirming action, device 300 selectsaffordance 306. That is, affordance 306 is selected in response to thecombination of the user looking at affordance 306 and providing aconfirming action. The confirming action is beneficial for preventingfalse positives (e.g., incorrect determinations by device 300 that user200 desires to select or act upon affordance 306). Non-limiting examplesof a confirming action include an eye gesture, a body gesture, a voiceinput, a controller input, or a combination thereof.

Examples of an eye gesture include a single blink, multiple blinks, apredetermined number of blinks, a predetermined number of blinks withina predetermined amount of time, a blink of a predetermined duration(e.g., eyes closed for one second), a blink pattern (e.g., one slowblink followed by two rapid blinks), a wink, a wink with a particulareye, a wink pattern (e.g., left, right, left, each with a specifiedduration), an predetermined eye motion (e.g., a quick look up), a “long”look or dwell (e.g., continuously maintaining the gaze direction in thedirection of affordance 306 (or in a direction corresponding toaffordance 306) for a predetermined amount of time), or an eye motionthat meets some other predetermined criteria.

Examples of a hand gesture include placement of a hand at a locationcorresponding to the location of affordance 306 (e.g., between the userand the display of affordance 306), a wave, a pointing motion (e.g., ataffordance 306), or a gesture with a predefined motion pattern. In someembodiments, the hand gesture confirming action depends on the locationof the hand gesture (e.g., the hand gesture must be at a particularlocation). In some embodiments, the hand gesture confirming action doesnot depend on the location of the hand gesture (e.g., the hand gestureis location-independent).

Example of a voice input include a voice command (e.g., “pick that up”or “turn on the light”). In some embodiments, the voice input explicitlyidentifies the object associated with affordance 306 (e.g., “select thebox”). In some embodiments, the voice input does not explicitly identifythe object associated with the affordance and instead refers to theobject using a pronoun that is otherwise ambiguous (e.g., “grab that”).

With respect to a controller input, in some embodiments, device 300 isin communication with a controller that is configured to receive inputsvia, for example, a button, trigger, joystick, scroll wheel, knob,keyboard, or touch-sensitive surface (e.g., a touchpad ortouch-sensitive display). In some embodiments, the controller and device300 are connected wirelessly or via a wired connection. Examples of acontroller input include a press of a button, a pull of a trigger, amovement of a joystick, a rotation of a scroll wheel, a rotation of aknob, a press of a button on a keyboard, or a contact or gesture (e.g.,tap or swipe) on a touch-sensitive surface.

In some embodiments, selecting affordance 306 includes applying focus onaffordance 306. Optionally, device 300 provides an indication thataffordance 306 has been selected. In some embodiments, the indicationincludes an audio output (e.g., a beep), a visual indication (e.g.,outlining or highlighting the selected affordance), or haptic output.Optionally, affordance 306 remains selected for a predetermined amountof time (e.g., focus is maintained on affordance 306 for thepredetermined amount of time). Optionally, affordance 306 remainsselected until a deselection input is received. In some embodiments, thedeselection input is the same input as the confirming action. In someembodiments, the deselection input is a different input than theconfirming action. In some embodiments, the deselection input includesan eye gesture, a body gesture, a voice input, a controller input, or acombination or portion thereof, such as the exemplary inputs describedabove.

In some embodiments, affordance 306 remains selected until an actionassociated with affordance 306 (or the object with which it isassociated) is performed. FIG. 4 illustrates an exemplary actionperformed on affordance 306. While affordance 306 is selected, device300 receives an input (e.g., an eye gesture, a body gesture, a voiceinput, a controller input, or a combination or portion thereof, such asthe exemplary inputs described above). In the illustrated example, theinput includes user 200 changing the position of his eyes such that hisgaze direction moves on display 302 from location 308 to location 400shown in FIG. 4. In response to receiving the input, device 300 performsan action associated with affordance 306 in accordance with the input.In some embodiments, an action associated with affordance 306 isperformed in response to the input that causes device 300 to selectaffordance 306 (e.g., selecting affordance 306 includes performing theaction associated with affordance 306). In the example illustrated inFIG. 4, device 300 moves affordance 306 in accordance with the change inthe gaze direction of user 200, translating affordance 306 upward and tothe left on display 302 from the location of affordance 306 shown inFIG. 3 to the location shown in FIG. 4.

In addition to moving an affordance, exemplary actions includetransforming the affordance or a representation of an object associatedwith the affordance (e.g., rotating, twisting, stretching, compressing,enlarging, and/or shrinking affordance 306) and changing the state of adevice associated with the affordance (e.g., turning a lamp on or off).For example, in some embodiments, the affordance is a virtual dialassociated with a thermostat. A user can select the virtual dial andthen adjust the temperature of the thermostat. In some embodiments, someaspect of the position of an affordance (or object associated therewith)is automatically determined when an object is moved. For example, if avirtual picture frame that is initially lying flat on a horizontalsurface is moved to a wall, the frame is automatically rotated to avertical orientation to lay flat against the wall.

Turning now to FIG. 5, techniques related to resolving and selectingclosely spaced objects are described. FIG. 5 depicts virtual environment500 displayed on device 300. In some embodiments, environment 500 is aCGR environment (e.g., a VR or MR environment). Virtual environment 500includes affordance 502 and affordance 504, each associated with arespective box on top of virtual table 506 and displayed concurrently ondisplay 302. The dashed circle represents gaze direction 508 of user 200determined by device 300. The radius of the circle represents theangular uncertainty of gaze direction 508. As illustrated in FIG. 5,gaze direction 508 overlaps both affordance 502 and affordance 504,indicating that user 200 is interested in one of the affordances.Although gaze direction 508 is directed slightly more toward affordance502, the angular uncertainty of gaze direction 508 is greater than theangular separation between affordance 502 and affordance 504, whichprevents device 300 from determining with a sufficiently high level ofconfidence that gaze direction 508 corresponds to a specific one ofaffordance 502 and affordance 504. In other words, device 300 cannotresolve with sufficient confidence which affordance user 200 desires toselect. Instead, device 300 determines that gaze direction 508corresponds to both affordance 502 and affordance 504. In someembodiments in which a three-dimensional representation of a scene ispresented, a depth separation between the affordances may be less thanthe angular resolution or depth resolution of the gaze position.

In response to determining that gaze direction 508 corresponds to bothaffordance 502 and affordance 504, device 300 enlarges affordance 502and affordance 504. FIG. 6 illustrates affordance 502 and affordance 504after being enlarged (e.g., zoomed in). Affordance 502 and affordance504 appear as though they have been moved from the top of table 506 andpositioned closer to user 200. In FIG. 6, affordance 502 and affordance504 are enlarged by the same amount such that their relative size andposition remain the same (e.g., affordance 502 continues to appear to bein front of affordance 504). Zooming in affordance 502 and affordance504 increases the angular extent of affordance 502 and affordance 504and increases the angular separation between affordance 502 andaffordance 504. Optionally, the amount of zooming is based on the sizeof the affordances and/or the resolution of the gaze direction (e.g.,affordance 502 and affordance 504 are enlarged such that affordance 504is a predetermined minimum size). In some embodiments, affordance 502and affordance 504 are zoomed so that device 300 can resolve (with apredetermined level of confidence) which affordance user 200 isattempting to focus on.

In some embodiments, affordance 502 and affordance 504 are enlarged inaccordance with a determination that the gaze of user 200 meetspredefined criteria (e.g., gaze direction 508 corresponds to bothaffordance 502 and affordance 504 continuously for a predeterminedamount of time or for a predetermined amount of time within a predefinedwindow of time (e.g., 3 seconds during a 4 second window)). In someembodiments, affordance 502 and affordance 504 are enlarged in responseto device 300 receiving an input (e.g., an eye gesture, a hand gesture,a voice input, or a controller input as described above) while gazedirection 508 corresponds to both affordance 502 and affordance 504. Inthis way, user 200 can have improved control over when device zoomscontent. Also, in this way, device 300 can reduce or limit the instancesin which it invokes zooming functionality to resolve gaze ambiguities toinstances where resolution is necessary, thereby reducing strain on theuser and improving the user's experience. Optionally, the affordancescorresponding to the gaze direction are enlarged in accordance with theinput (e.g., a long and/or hard button press results in more enlargingthat a short and/or soft button press). In some embodiments, theaffordances are enlarged in accordance with a voice command (e.g., “zoomin 40%”). This allows user 200 to have increased control over thezooming.

Enlarging affordance 502 and affordance 504 provides user 200 with animproved view of the affordances and allows user 200 to more easily andconfidently select one of the affordances. For example, as shown in FIG.6, after affordance 502 and affordance 504 are enlarged, user 200decides that he wants to select affordance 502 and moves his line ofsight to gaze direction 510 on affordance 502. Notably, gaze direction510 no longer overlaps with affordance 504. Accordingly, device 300determines (e.g., with a relatively high degree of confidence) that gazedirection 510 corresponds to the direction of affordance 502 (and doesnot correspond to the direction of affordance 504). While gaze direction510 is determined to correspond to the direction of enlarged affordance502, user 200 selects affordance 502 with a confirming action, such asone of the confirming actions discussed above. Optionally, device 300performs an action associated with affordance 502 responsive to, and inaccordance with, the confirming input by user 200 for selecting enlargedaffordance 502 and/or in response to a further input while affordance502 is selected. In response to the confirming input by user 200 forselecting enlarged affordance 502, device 300 optionally reduces (e.g.,zooms out) affordance 502 and affordance 504 back to a previous state(e.g., the sizes and positions prior to being enlarge, depicted in FIG.5). In some embodiments, affordance 502 remains selected after beingreduced to the previous state.

In the embodiment described above with respect to FIGS. 5-6, device 300enlarges only affordances 502 and 504. In some embodiments, device 300displays an enlarged view of at least a portion of an environment thatsurrounds affordance 502 and affordance 504 in addition to enlargingaffordance 502 and affordance 504. FIG. 7 depicts an exemplaryembodiment in which device 300 determines a portion of virtualenvironment 500 that surrounds and includes the affordancescorresponding to gaze direction 508. The portion is designated byrectangle 700 and includes, for example, a portion of table 506 inaddition to affordances 502 and 504. As shown in FIG. 8, in response todetermining that gaze direction 508 corresponds to both affordance 502and affordance 504, device 300 enlarges the portion of virtualenvironment 500 designated by rectangle 700, including affordance 502and affordance 504. Although a portion of virtual environment 500 isenlarged along with in affordances 502 and 504, the affordances canstill be selected and acted upon as described above with respect toFIGS. 5-6. Furthermore, although the embodiments described above referto a virtual environment, similar techniques can be applied to other CGRenvironments, including mixed reality environments. For example, in someembodiments, a device includes a transparent display that displaysaffordances 502 and 504 overlaid on a user's live view of a physicalenvironment. The device also includes a user sensor for capturing usereye data and a scene sensor for capturing images of the physicalenvironment over which affordances 502 and 504 are displayed. Inresponse to determining that a user's gaze direction corresponds toaffordances 502 and 504, the exemplary device captures data of at leastthe physical environment surrounding affordances 502 and 504 anddisplays an enlarged representation (e.g., image) of the physicalenvironment surrounding affordances 502 and 504.

In the embodiments described above, affordances 502 and 504 aredisplayed in a two-dimensional representation of a virtual environment.In some embodiments, affordances are displayed in a three-dimensional(3D) representation of an environment on, for example, virtual realityHMD 100 a depicted in FIGS. 1F-1H. FIG. 9 illustrates a 3Drepresentation of virtual environment 902 displayed on HMD 900. In someembodiments, environment 902 is a CGR environment (e.g., a VR or MRenvironment). Virtual environment 902 includes affordance 904 andaffordance 906. Affordance 904 has a first depth and affordance 906 hasa second depth that is greater than the first depth of affordance 904.Since virtual environment 902 is a 3D representation, device 900determines a gaze position, which in the illustrated embodiment includesa gaze direction and a gaze depth, based on data captured from the eyesof the user. In some embodiments, determining a gaze position includesdetermining a gaze direction, but not necessarily a gaze depth. In someembodiment, determining a gaze position includes determining a gazedepth, but not necessarily a gaze direction.

In FIG. 9, the radius of the cylinder surrounding gaze position 908represents the angular resolution of the gaze direction, and the lengthof the cylinder represents the depth resolution of the gaze depth (e.g.,the uncertainty in the gaze depth). Based on the gaze direction, angularresolution, gaze depth, and depth resolution, device 900 determineswhether the position of affordance 904 and/or affordance 906 correspondto the gaze position. In some embodiments, device 900 determines whetherposition of affordance 904 and/or affordance 906 correspond to the gazeposition based on the gaze direction (and optionally the angularresolution) regardless of the gaze depth, or based on the gaze depth(and optionally the depth resolution) regardless of the gaze direction.

In some embodiments, device 900 enhances the display of the more distantaffordance (e.g., affordance 906) in response to determining that thegaze position corresponds to both affordance 904 and affordance 906.According to the embodiment depicted in FIG. 10, affordance 906 isenhanced by making affordance 906 brighter relative to affordance 904(e.g., by increasing the brightness of affordance 906, decreasing thebrightness of affordance 904, or a combination of both). In someembodiments, enhancing an affordance includes altering the visualappearance of the affordance itself (e.g., by making the affordancebrighter or changing the color of the affordance). In some embodiments,enhancing an affordance includes degrading the visual appearance ofother aspects of an environment (e.g., by making another affordance orthe surrounding environment appear blurry). Similarly, in a 2Drepresentation of a 3D environment, a smaller object or an object thathas a greater depth value in the 3D environment is optionally enhanced.

In some embodiments, in response to device 900 determining that gazeposition 908 corresponds to both affordance 904 and affordance 906(e.g., that device 900 cannot resolve which affordance the user islooking at), device 900 enlarges affordance 904 and affordance 906. Insome embodiments that provide a 3D representation, an affordance isenlarged from the perspective of the user by moving the affordancetoward the user and displaying the affordance at a depth that appearscloser to the user. FIG. 11 depicts an embodiment similar to theembodiment depicted in FIG. 6, in which affordance 904 and affordance906 are enlarged (e.g., moved closer to the user) while maintainingtheir relative size and position. FIG. 12 depicts an embodiment in whichaffordance 904 and affordance 906 are enlarged and re-positionedrelative to each other such that affordance 904 and affordance 906 aredisplayed side by side at the same depth. It should be recognized thatan analogous technique can also be applied to a 2D representation of anenvironment. For example, when gaze direction 508 is determined tocorrespond to a gaze at both affordance 502 and affordance 504 asdescribed above with respect to FIG. 5, affordance 502 and affordance504 are optionally enlarged by different amounts relative to each other,and/or re-positioned relative to each other such that affordance 502 andaffordance 504 are displayed side-by-side. Furthermore, when affordance904 and affordance 906 are enlarged, device 900 can further determinewhether an updated gaze position corresponds to one of the enlargedaffordances and select and/or perform an action on an affordance in amanner analogous to the techniques described previously with respect toFIGS. 3-4 and 6-8.

Turning now to FIG. 13, techniques for altering the display of objectsbased on object depth are described. FIG. 13 again depicts device 300.Device 300 displays environment 1300 (e.g., a CGR environment) withobject 1302 and object 1304 concurrently displayed. As depicted in FIG.13, object 1302 appears closer (e.g., has a smaller depth value) thanobject 1304. Also, from the perspective displayed in FIG. 13, object1302 partially obstructs the view of object 1304. Gaze position 1306 islocated on object 1302. Gaze position 1306 optionally includes eithergaze direction or gaze depth, or both. Device 300 optionally determineswhether gaze position corresponds to object 1302 and/or object 1304based on either gaze direction or gaze depth, or both, according to anyof the techniques described above.

Device 300 visually alters object 1302 and/or object 1304 based onwhether gaze position 1306 corresponds to object 1302 or object 1304.Device 300 determines whether gaze position 1306 corresponds to object1302 or object 1304 according to any of the techniques described above.In some embodiments, in response to determining that gaze position 1306corresponds to object 1302, device 300 visually alters the display ofobject 1304; and in response to determining that gaze position 1306corresponds to object 1304, device 300 visually alters the display ofobject 1302. For example, if the user's focus, as determined by eitherdirection or depth, or both, is determined to be on one of the objects,the visual appearance of the other object is altered in order toemphasize the object of the user's focus. As depicted in FIG. 14, device300 determines that gaze position 1306 corresponds to object 1302, andin response, visually alters object 1304 in a way that emphasizes object1302 and/or de-emphasizes object 1304. Examples of visually altering anobject that de-emphasizes the object include making the object appearblurry or fuzzy, decreasing the resolution of the object, decreasing thebrightness of the object, decreasing the contrast of the object,increasing the transparency of the object, and ceasing to display theobject. In some embodiments, device 300 visually alters object 1302 orobject 1304 in response to receiving an input (e.g., an eye gesture, ahand gesture, a voice input, or a controller input) and determining thatgaze position 1306 corresponds to object 1302 or object 1304,respectively. Optionally, device 300 alters object 1302 and object 1304in accordance with a determination that the directions of both objectscorrespond to the gaze direction, indicating that one of the objects islikely obstructing the other and that distinguishing the objects wouldbe advantageous.

Optionally, device 300 also visually alters the display of the objectcorresponding to the gaze position (e.g., object 1302) to enhance theappearance of the object. Examples of visually enhancing an objectinclude making the object appear sharper, increasing the resolution ofthe object, increasing the brightness of the object, increasing thecontrast of the object, decreasing the transparency of the object,highlighting the object, and causing the object to appear.

In FIG. 15, the user has moved his gaze position from 1306 to location1500 corresponding to object 1304. In response, device 300 visuallyalters object 1302 and returns object 1304 to the appearance initiallydisplayed in FIG. 13. In the embodiment illustrated in FIG. 15, device300 makes object 1302 semi-transparent so that the user can better seethe object on which he is attempting to focus. Optionally, device 300removes object 1302 to provide an unobstructed view of object 1304.

It should be recognized that the embodiments discussed above withrespect to FIGS. 2-15 are exemplary and are not intended to be limiting.For example, although the embodiments in FIGS. 2-12 are described withrespect to a virtual environment, the techniques can be appliedanalogously to other CGR environments, including mixed realityenvironments.

Turning now to FIG. 16, a flow chart of exemplary process 1600 forinteracting with an electronic device using an eye gaze is depicted.Process 1600 can be performed using a user device (e.g., 100 a, 300, or900). The user device is, for example, a handheld mobile device, ahead-mounted device, or a head-up device. In some embodiments, process1600 is performed using two or more electronic devices, such as a userdevice that is communicatively coupled to another device, such as a basedevice. In these embodiments, the operations of process 1600 aredistributed in any manner between the user device and the other device.Further, the display of the user device can be transparent or opaque.Process 1600 can be applied to CGR environments, including virtualreality and mixed reality environments, and to affordances thatcorrespond to virtual objects or physical objects. Although the blocksof process 1600 are depicted in a particular order in FIG. 16, theseblocks can be performed in other orders. Further, one or more blocks ofprocess 1600 can be partially performed, optionally performed, combinedwith another block(s), and/or additional blocks can be performed.

At block 1602, the device displays an affordance associated with a firstobject (e.g., a displayed object).

At block 1604, the device determines a gaze direction or a gaze depth(e.g., of one or more eyes). In some embodiments, data is captured froma sensor directed toward the user, and the gaze direction or the gazedepth is determined based on the data captured from the sensor. In someembodiments, determining the gaze direction or the gaze depth includesdetermining the gaze direction. In some embodiments, determining thegaze direction or the gaze depth includes determining the gaze depth.Optionally, the gaze direction or the gaze depth is determined using raycasting or cone casting. Optionally, an angular extent of a cone usedfor the cone casting is based on the angular resolution of the gazedirection.

At block 1606, the device determines whether the gaze direction or thegaze depth corresponds to a gaze at the affordance. In some embodiments,determining that the gaze direction or the gaze depth corresponds to thedepth of the affordance includes determining that the gaze is directedto the affordance. In some embodiments, determining that the gaze isdirected to the affordance is based at least in part on the angularresolution of the gaze direction. In some embodiments, determining thatthe gaze direction or the gaze depth corresponds to a gaze at theaffordance includes determining that the gaze depth corresponds to thedepth of the affordance. In some embodiments, determining that the gazedepth corresponds to the depth of the affordance is based at least inpart on the depth resolution of the gaze depth.

At block 1608, while the gaze direction or the gaze depth is determinedto correspond to a gaze at the affordance, the device receives a firstinput representing an instruction to take action on the affordancecorresponding to the first object. In some embodiments, the first inputincludes an eye gesture, a hand gesture, a voice input, and/or acontroller input.

At block 1610, the device selects the affordance responsive to receivingthe first input. Optionally, while the affordance is selected, a secondinput is received and an action associated with the selected affordanceis performed in response to receiving the second input and in accordancewith the second input. In some embodiments, second input includes an eyegesture, a hand gesture, a voice input, or an input on a controller.

Turning now to FIG. 17, a flow chart of exemplary process 1700 forinteracting with an electronic device using an eye gaze is depicted.Process 1700 can be performed using a user device (e.g., 100 a, 300, or900). The user device is, for example, a handheld mobile device, ahead-mounted device, or a head-up device. In some embodiments, process1700 is performed using two or more electronic devices, such as a userdevice that is communicatively coupled to another device, such as a basedevice. In these embodiments, the operations of process 1700 aredistributed in any manner between the user device and the other device.Further, the display of the user device can be transparent or opaque.Process 1700 can be applied to CGR environments, including virtualreality and mixed reality environments, and to affordances thatcorrespond to virtual objects or physical objects. Although the blocksof process 1700 are depicted in a particular order in FIG. 17, theseblocks can be performed in other orders. Further, one or more blocks ofprocess 1700 can be partially performed, optionally performed, combinedwith another block(s), and/or additional blocks can be performed.

At block 1702, the device displays a first affordance and a secondaffordance. Optionally, the first affordance and the second affordanceare displayed concurrently. In some embodiments, the first affordanceand the second affordance are displayed with a two-dimensionalrepresentation or a three-dimensional representation of an environment(e.g., a CGR environment) that includes the first affordance and thesecond affordance. Optionally, the first affordance is displayed at afirst depth in a three-dimensional representation of the environment andthe second affordance is displayed at a second depth in thethree-dimensional representation of the environment, where the firstdepth is different than the second depth.

At block 1704, the device determines a first gaze direction or a firstgaze depth (e.g., of one or more eyes). In some embodiments, data iscaptured from a sensor directed toward the user, and the gaze directionor the gaze depth is determined based on the data captured from thesensor. Optionally, gaze direction or the gaze depth is determined usingray casting or cone casting. In some embodiments, an angular extent of acone used for the cone casting is based on an angular resolution of thegaze direction.

At block 1706, the device determines whether the first gaze direction orthe first gaze depth corresponds to a gaze at both the first affordanceand the second affordance. Optionally, in response to determining thatthe first gaze direction or the first gaze depth corresponds to both thefirst affordance and the second affordance, the display of the firstaffordance is enhanced in accordance with the first depth being greaterthan the second depth; and the display of the second affordance isenhanced in accordance with the second depth being greater than thefirst depth. In some embodiments, the gaze direction is determined, anddetermining that the gaze direction or the gaze depth corresponds toboth the first affordance and the second affordance includes determiningthat the gaze direction corresponds to both the first affordance and thesecond affordance. Optionally, determining that the gaze directioncorresponds to both the first affordance and the second affordance isbased at least in part on the angular resolution of the gaze direction.In some embodiments, determining the gaze direction or the gaze depthincludes determining the gaze depth, and determining that the gazedirection or the gaze depth corresponds to both the first affordance andthe second affordance includes determining that the gaze depthcorresponds to both the first affordance and the second affordance.Optionally, determining that the gaze depth corresponds to both thefirst affordance and the second affordance is based at least in part onthe depth resolution of the gaze depth.

At block 1708, the device enlarges the first affordance and the secondaffordance responsive to determining that the first gaze direction orthe first gaze depth corresponds to a gaze at both the first affordanceand the second affordance. In some embodiments, the first affordance andthe second affordance are enlarged in accordance with a determinationthat the user's gaze meets predefined criteria. In some embodiments, athird input is received, and the first affordance and the secondaffordance are enlarged in response to determining that the first gazedirection or the first gaze depth corresponds to both the firstaffordance and the second affordance and receiving the third input. Insome embodiments, the third input includes an eye gesture, a handgesture, a voice input, or a controller input. In some embodiments,enlarging the first affordance and the second affordance includesdisplaying an enlarged view of at least a portion of an environment(e.g., a CGR environment) that surrounds the first affordance and thesecond affordance. In some embodiments, the enlarged view of the atleast a portion of the environment that surrounds the first affordanceand the second affordance is a representation of a virtual environment.In some embodiments, the enlarged view of the at least a portion of theenvironment that surrounds the first affordance and the secondaffordance is a representation of a physical environment. In someembodiments, enlarging the first affordance and the second affordanceincludes displaying the first affordance at a third depth in athree-dimensional representation of the environment and displaying thesecond affordance at a fourth depth in the three-dimensionalrepresentation of the environment, where the third depth is the same asthe fourth depth.

Optionally, after enlarging the first affordance and the secondaffordance, a second gaze direction or a second gaze depth isdetermined, and the second gaze direction or the second gaze depth isdetermined to correspond to a gaze at the first affordance. While thesecond gaze direction or the second gaze depth is determined tocorrespond to a gaze at the first affordance, a first input representinguser instruction to take action on the first affordance is received, andthe first affordance is selected responsive to receiving the firstinput. Optionally, the first input includes an eye gesture, a handgesture, a voice input, or a controller input.

In some embodiments, the first affordance or the second affordance isreduced in response to receiving the first input. Optionally, while thefirst affordance is selected, a second input is received, and an actionassociated with the first affordance in accordance with the second inputis performed in response to receiving the second input. In someembodiments, the second input includes an eye gesture, a hand gesture, avoice input, or a controller input.

Turning now to FIG. 18, a flow chart of exemplary process 1800 forinteracting with an electronic device using an eye gaze is depicted.Process 1800 can be performed using a user device (e.g., 100 a, 300, or900). The user device is, for example, a handheld mobile device, ahead-mounted device, or a head-up device. In some embodiments, process1800 is performed using two or more electronic devices, such as a userdevice that is communicatively coupled to another device, such as a basedevice. In these embodiments, the operations of process 1800 aredistributed in any manner between the user device and the other device.Further, the display of the user device can be transparent or opaque.Process 1800 can be applied to CGR environments, including virtualreality and mixed reality environments, and to virtual objects, physicalobjects, and representations thereof. Although the blocks of process1800 are depicted in a particular order in FIG. 18, these blocks can beperformed in other orders. Further, one or more blocks of process 1800can be partially performed, optionally performed, combined with anotherblock(s), and/or additional blocks can be performed.

The device is adapted to display a field of view of a three-dimensionalcomputer generated reality environment. The field of view is renderedfrom a viewing perspective. At block 1802, the device displays a firstobject and a second object. Optionally, the first object and the secondobject are displayed concurrently. In some embodiments, the first objectand the second object are displayed such that the first object appearsto be (e.g., is presented as) closer than the second object from theviewing perspective.

At block, 1804, the device determines a gaze position (e.g., of one ormore eyes). In some embodiments, data is captured from a sensor directedtoward the user, and the gaze position is determined based on the datacaptured from the sensor. In some embodiments, the gaze position isdetermined using ray casting or cone casting. Optionally, an angularextent of a cone used for the cone casting is based on the angularresolution of a gaze direction.

At block 1806, the device determines whether the gaze positioncorresponds to a gaze at the first object or the second object. In someembodiments, the gaze direction is determined, and determining that thegaze position corresponds to a gaze at the first object or the secondobject includes determining that the gaze is directed at the firstobject or the second object. Optionally, determining whether the gaze isdirected to the first object or the second object is based at least inpart on the angular resolution of the gaze direction. In someembodiments, the gaze depth is determined, and determining that the gazeposition corresponds to a gaze at the first object or the second objectincludes determining that the gaze depth corresponds to a depth of thefirst object or the second object (e.g., as presented in the field ofview). Optionally, determining that the gaze depth corresponds to thedepth of the first object or the second object is based at least in parton the depth resolution of the gaze depth.

At block 1808, the device visually alters the display of the secondobject in accordance with a determination that the gaze positioncorresponds to a gaze at the first object. In some embodiments, thesecond object is altered in response to determining that the gazeposition corresponds to a gaze at the first object and receiving aninput. The input optionally includes an eye gesture, a hand gesture, avoice input, or a controller input. Optionally, the device enhances thedisplay (e.g., display resolution) of the first object. Optionally,after visually altering the second object, the device determines asecond gaze position of the user, and in accordance with a determinationthat the second gaze position corresponds to a gaze at the secondobject, visually alters the display of the first object and displays thesecond object according to its initial appearance.

At block 1810, the device visually altering the display of the firstobject in accordance with a determination that the gaze positioncorresponds to a gaze at the second object. In some embodiments, thefirst object is altered in response to determining that the gazeposition corresponds to a gaze at the second object and receiving aninput. The input optionally includes an eye gesture, a hand gesture, avoice input, or a controller input. Optionally, the device enhances thedisplay (e.g., display resolution) of the second object.

Executable instructions for performing the features of methods 1600,1700, and/or 1800 described above are, optionally, included in atransitory or non-transitory computer-readable storage medium (e.g.,memory(ies) 106) or other computer program product configured forexecution by one or more processors (e.g., processor(s) 102). Further,some operations in method 1600 (e.g., block 1610) are, optionally,included in method 1700 and/or method 1800, some operations in method1700 (e.g., block 1708) are, optionally, included in method 1600 and/ormethod 1800, and some operations in method 1800 (e.g., blocks 1806,1808, and/or 1810) are, optionally, included in method 1600 and/ormethod 1700.

Turning to FIGS. 19A-19Y, techniques are described that provide adual-modality for, e.g., selecting and/or placing objects (e.g., virtualobjects, physical objects, and affordances corresponding to virtual andphysical objects) in a CGR environment. In a first mode (e.g.,“gaze-engaged” mode), a position or object is initially designated basedon the position of a user's gaze. After the initial designation, asecond mode (e.g., “gaze-disengaged” mode) is used to move thedesignated position or designate a different object without using gaze.Gaze position can be rapidly moved by a user, which makes it effectivefor quickly identifying a general area. As discussed above, however,there is uncertainty in the position of a user's gaze, which makes itdifficult to designate a precise position using gaze. Displaying apersistent visual indicator at the estimated gaze position may beineffective for designating a precise position, as the indicator candistract the user and cause the user's gaze to follow the indicatorrather than focus on the desired designation point. The dual-modalitytechnique allows a user to quickly make a rough initial designation, andthen make fine adjustments independent of gaze (e.g., based solely onmanual inputs) to designate a particular point or object.

FIG. 19A illustrates user 200, who uses device 1900 to interact withvirtual environment 1902. In some embodiments, environment 1902 is a CGRenvironment (e.g., a VR or MR environment). Device 1900 includes virtualreality HMD 1900 a and input device 1900 b. In some embodiments, HMD1900 a is device 100 a (e.g., in FIGS. 1F-1I) and input device 1900 b isin communication with HMD 1900 a (e.g., via communication bus(es) 150,shown in FIGS. 1A-1B). View 1902 a illustrates a view of virtualenvironment 1902 displayed to user 200 on HMD 1900 a, and view 1902 billustrates a perspective view of virtual environment 1902 includinguser 200. FIG. 19A also illustrates input device 1900 b, which includesa touch-sensitive surface 1904 (e.g., touch-sensitive surface 122 inFIGS. 1A-1B) that allows user 200 to provide inputs to interact withvirtual environment 1902. Device 1900 includes sensor(s) (e.g., imagesensor(s) on HMD 1900A) to determine gaze 1906 (e.g., gaze directionand/or gaze depth) of user 200 (as described above). In someembodiments, device 1900 includes sensor(s) configured to detect varioustypes of user inputs, including (but not limited to) eye gestures, bodygestures, and voice inputs. In some embodiments, input device includes acontroller configured to receive button inputs (e.g., up, down, left,right, enter, etc.).

Virtual environment 1902 includes stack of photos 1908, which includesindividual photos 1908 a-1908 e, lying on table 1912. Gaze 1906 seen inview 1902 b indicates that user 200 is looking at stack of photos 1908.In some embodiments, the lines representing gaze 1906 are not visible invirtual environment 1902, as shown for example in view 1902 a.

As shown in FIG. 19A, device 1900 receives user input 1910 a (e.g., atouch gesture on touch-sensitive surface 1904) while gaze 1906 isdirected at stack of photos 1908. In some embodiments, user input 1910 aincludes an eye gesture, a body gesture, a voice input, a controllerinput, or a combination thereof, in addition to or instead of a touchgesture on touch-sensitive surface 1904.

In some embodiments, the response to user input 1910 a depends on thecharacteristics of user input 1910 a. For example, in accordance with adetermination that user input 1910 a is a first type of input (e.g., atap on touch-sensitive surface 1904), the entire stack of photos 1908 isselected, as indicated by focus indicator 1914 (e.g., bold border)around stack of photos 1908 in FIG. 19B. In some embodiments, device1900 de-selects stack of photos 1908 in response to receiving furtherinput (e.g., selection of an exit button).

Alternatively, in accordance with a determination that user input 1910 ais a different type of input (e.g., a touch and hold on touch-sensitivesurface 1904), photos 1908 a-1908 e are presented, as illustrated inFIG. 19C, so that user 200 can more easily select a particular photofrom stack 1908. In FIG. 19C, photos 1908 a-1908 e are moved from table1912 and presented upright and spread out in the middle of the field ofview of user 200. In response to receiving user input 1910 a, photo 1908a in the far left position is designated (e.g., tentatively selected).Designation of photo 1908 a is indicated by focus indicator 1914, whichincludes a bold border around photo 1908 a. In some embodiments, focusindicator 1914 includes a pointer, cursor, dot, sphere, highlighting,outline, or ghost image that visually identifies the designated object.In some embodiments, device 1900 un-designates photo 1908 a and returnsphotos 1908 to table 1912 in response to receiving further input (e.g.,selection of an exit button or liftoff of a touch).

The responses illustrated in FIG. 19B and FIG. 19C are both based ongaze 1906, and more specifically, the gaze position of user 200 at thetime of user input 1910 a. Stack of photos 1908 is either selected (FIG.19B) or designated and re-presented for further selection due to thegaze position of user 200 being positioned on stack of photos 1908. Whenthe response to user input is based on gaze 1906, device 1900 is in agaze-engaged mode and gaze 1906 of user 200 is engaged with user input.Gaze-engaged mode is indicated in FIG. 19A by gaze 1906 beingillustrated by solid lines.

In some embodiments, the response to user input 1910 a depends onwhether gaze 1906 corresponds to more than one selectable object. Insome embodiments, device 1900 does not confirm selection if there isambiguity or uncertainty about the object to be selected. For example,device 1900 displays photos 1908 a-1908 e and designates photo 1908 a(FIG. 19C) in accordance with a determination that the position of gaze1906 corresponds to a plurality of unresolvable selectable objects(e.g., stack of photos 1908). In some such embodiments, in accordancewith a determination that the position of gaze 1906 corresponds to onlya single selectable object (e.g., mug 1918 illustrated in FIG. 19Mdescribed below), device 1900 selects the single selectable object(e.g., instead of designating an object or providing the capability tofurther refine the selection).

In the illustrated embodiment, in response to receiving user input 1910a, device 1900 also switches to a gaze-disengaged mode in which aresponse to a user input is not based on gaze 1906 of user 200 and gaze1906 is disengaged from further user input. Gaze-disengaged mode isindicated in FIG. 19C by gaze 1906 being illustrated by broken lines.

Turning to FIG. 19D, while photo 1908 a is designated, device 1900receives user input 1910 b. In FIG. 19D, user input 1910 b includes aleft-to-right swipe or drag gesture. In some embodiments, user input1910 b is a continuation of user input 1910 a (e.g., user input 1910 aincludes a contact that is maintained on touch-sensitive surface 1904,and user input 1910 b includes movement of the contact). In someembodiments, user input 1910 b includes a press of a directional buttonor an oral command (“move right”). In response to receiving user input1910 b, focus indicator 1914 is moved from photo 1908 a in accordancewith (e.g., in the direction of) user input 1910 b to designate photo1908 b, as shown in FIG. 19E.

Notably, since gaze 1906 is disengaged, photo 1908 b is designated inresponse to receiving user input 1910 b despite gaze 1906 beingpositioned on photo 1908 a at the time of user input 1910 b. Focusindicator 1914 is moved to a position (e.g., object) that does notcorrespond to the position of gaze 1906. More generally, moving focusindicator 1914 to designate photo 1908 b is not based on gaze 1906. Insome embodiments, focus indicator 1914 is moved based solely oncharacteristics of user input 1910 b (e.g., position, direction, speed,duration, etc.).

As illustrated in FIG. 19E, gaze 1906 remains disengaged, and inresponse to receiving further user input 1910 c while gaze 1906 ispositioned on photo 1908 a, focus indicator 1914 is moved from photo1908 b to designate photo 1908 c, as shown in FIG. 19F.

Turning to FIG. 19G, while photo 1908 c is designated, device 1900receives user input 1910 d (e.g., click, double tap, or finger liftoff).In response to receiving user input 1910 d, the currently designatedobject, photo 1908 c, is selected. In response to receiving user input1910 d, focus indicator 1914 remains on photo 1908 c and the otherphotos 1908 a, 1908 b, 1908 d, and 1908 e are returned to table 1912, asshown in FIG. 19H. Also, gaze 1906 of user 200 is re-engaged in responseto receiving user input 1910 d.

The technique described with respect to FIGS. 19A-19H provides adual-mode operation in which user 200 can efficiently select aparticular object (e.g., one of photos 1908) that would be difficult todistinguish using gaze alone when photos 1908 are stacked on table 1912(e.g., due to uncertainty in gaze position). User 200 can use gaze 1906to quickly designate a group of objects, and then use inputs independentfrom gaze 1906 to navigate the group of objects and select a particularone.

Turning to FIG. 19I, while maintaining selection of photo 1908 c, user200 moves gaze 1906 to a position on wall 1916 in environment 1902between photo 1908 f and photo 1908 g. In response to movement of gaze1906, photo 1908 c is moved to the position corresponding to the gazeposition. In some embodiments, photo 1908 c remains at the positionshown in FIG. 19I or is moved and/or modified visually (e.g., so as notto obstruct the view of user 200 of virtual environment 1902) until aplacement position for photo 1908 c is designated or selected, asdescribed below.

While photo 1908 c is positioned as shown in FIG. 19I, device 1900receives user input 1910 e (e.g., a touch on touch-sensitive surface1904). In response to receiving user input 1910 e, a placement positionfor photo 1908 c is designated based on the position of gaze 1906 at thetime of user input 1910 e. As shown in FIG. 19J, in response toreceiving user input 1910 e, selected photo 1908 c is placed at theposition of gaze 1906 and remains selected, and gaze 1906 is disengaged.In some embodiments, the placement position is indicated by a pointer,cursor, dot, sphere, highlighting, outline, or ghost image (e.g., of theobject being placed).

In some embodiments, the response to user input 1910 e depends on thecharacteristics of user input 1910 e. In some embodiments, in accordancewith user input 1910 e including a first type of input (e.g., a touch ontouch-sensitive surface 1904), device 1900 designates a tentativeplacement position for photo 1908 c on wall 1916, photo 1908 c remainsselected, and gaze 1906 is disengaged, as discussed above; and inaccordance with user input 1910 e including a second type of input(e.g., a click on touch-sensitive surface 1904), photo 1908 c is placedon wall 1916, photo 1908 c is de-selected, and gaze 1906 is re-engaged.Accordingly, by using different inputs, user 200 can choose to eitherdesignate a tentative placement position and maintain selection of photo1908 c to adjust the position with further input (as discussed below) oraccept the gaze position as the placement position and de-select photo1908 c.

Returning to FIG. 19J, while photo 1908 c remains selected andpositioned at the initially designated position, device 1900 receivesuser input 1910 f, which includes a downward swipe or drag gesture. Inresponse to receiving user input 1910 f, photo 1908 c is moved downwardin accordance with user input 1910 f and independent of the position ofgaze 1906, as shown in FIG. 19K. According to this technique, user 200can use gaze 1906 to quickly and roughly designate an initial placementposition and then make fine adjustments to the position that do notdepend on gaze. Once photo 1908 c is in the desired position (e.g.,aligned with photo 1908 f and photo 1908 g), user 200 provides input1910 g. In response to input 1910 g, the position of photo 1908 c inFIG. 19K is selected as the final placement position and gaze 1906 isre-engaged. As shown in FIG. 19L, in response to receiving user input1910 g, focus is removed from photo 1908 c (photo 1908 c is de-selected)and photo 1908 c remains at the selected placement position as user 200moves gaze 1906 to a different position.

Turning now to FIG. 19M, virtual environment 1902 is configured as shownin FIG. 19L, with the addition of mug 1918. In FIG. 19M, while gaze 1906is positioned between mug 1918 and stack of photos 1908, device 1900receives user input 1910 h (e.g., a touch on touch-sensitive surface1904, a press of a button, or a body gesture). In response to receivinguser input 1910 h, a selection point represented by focus indicator 1920is designated at a position corresponding to the position of gaze 1906,as shown in FIG. 19N. In some embodiments, focus indicator 1920 includesa pointer, cursor, dot, or sphere. In some embodiments, gaze 1906 isdetermined (e.g., measured or estimated) prior to the user input (e.g.,the most recent measured or estimated position) or subsequent (e.g., inresponse) to the user input.

In some embodiments, the response to user input 1910 h is contextual. Insome embodiments, the response is based on what is located at the gazeposition. For example, device 1900 can respond differently depending onwhether an object, a plurality of unresolvable objects, a menuaffordance, or no object is at the gaze position at the time of userinput 1910 h. For example, if device 1900 determines with apredetermined amount of certainty that gaze 1906 corresponds to stack ofphotos 1908, focus indicator 1914 is displayed as described withreference to FIGS. 19A-19C instead of focus indicator 1920. In someembodiments, a plurality of objects include menu options associated withan object (e.g., menu affordance 1924 described below).

In some embodiments, the response is based on whether an object iscurrently selected. For example, if an object is not currently selected,device 1900 can operate in a selection mode and perform a selectionaction (e.g., select an object (FIG. 19B), designate a selection point(FIG. 19N), or display multiple objects to select (FIG. 19C)); if anobject is currently selected, device 1900 can operate in a placementmode and perform a placement action (e.g., place the object at the gazeposition or display a ghost image of the selected object at a designatedplacement position; see, e.g., the description of FIGS. 19H-19J aboveand FIGS. 19P-19Q below).

Returning to FIG. 19N, focus indicator 1920 is displayed in accordancewith a determination that there are multiple selectable objectscorresponding to the position of gaze 1906 at the time of user input1910 h. For example, if device 1900 is unable to determine withsufficient certainty whether gaze 1906 corresponds to stack of photos1908 or mug 1918, focus indicator 1920 is displayed so that user 200 canclarify which object he desires to select. In the illustratedembodiment, user 200 desires to select mug 1918 and provides user input1910 i, which includes a right-to-left swipe or drag gesture ontouch-sensitive surface 1904 to move focus indicator 1920 to mug 1918.In response to receiving user input 1910 i, the selection point is movedto a position corresponding to mug 1918, as indicated by focus indicator1920 in FIG. 19O. While the selection point is positioned on mug 1918,device 1900 receives user input 1910 j (e.g., a click). In response toreceiving user input 1910 j, the selection point is confirmed and theobject corresponding to the current position of the selection point isselected. As illustrated in FIG. 19P, mug 1918 is selected, as indicatedby focus indicator 1915 (e.g., highlighting) around mug 1918, and gaze1906 is re-engaged.

As shown in FIG. 19P, while mug 1918 remains selected, user 200 movesgaze 1906 to table 1922. In the illustrated embodiment, mug 1918 remainsdisplayed at the same position (e.g., it does not move with gaze 1906even though gaze 1906 is engaged).

While gaze 1906 is positioned as shown in FIG. 19P, device 1900 receivesuser input 1910 k. In response to receiving user input 1910 k, aplacement point is designated by focus indicator 1920 at the positioncorresponding to gaze 1906, and gaze 1906 is disengaged, as shown inFIG. 19Q. In some embodiments, focus indicator 1920 includes a pointer,cursor, dot, sphere, highlighting, outline, or ghost image of theselected object (e.g., mug 1918).

In some embodiments, the response to user input 1910 k, while mug 1918is selected, depends on whether there is more than one placementposition corresponding to the position of gaze 1906 at the time of userinput 1910 k. In FIG. 19P, the placement point is designated inaccordance with a determination that the position of gaze 1906corresponds to a plurality of possible selectable placement positions(e.g., device 1900 does not confirm the placement position for theselected object if there are various possible placement position at ornear the gaze position). In some embodiments, in accordance with adetermination that the position of gaze 1906 corresponds to only asingle selectable placement position and in response to receiving userinput 1910 k, device 1900 places the selected object at the gazeposition, de-selects the object, and re-engages gaze 1906.

Turning to FIG. 19Q, while the placement point is designated, device1900 receives user input 1910 l, which includes a diagonal swipe or draggesture upward and to the right on touch-sensitive surface 1904. Inresponse to receiving user input 1910 l, the placement point is moved inaccordance with user input 1910 l, as indicated by focus indicator 1920moving toward the center of table 1922, as shown in FIG. 19R. Theplacement point is confirmed in response to receiving user input 1910 m,and the selected object (e.g., mug 1918) is placed at the confirmedplacement point on table 1922, as shown in FIG. 19S. Gaze 1906 is alsore-engaged in response to receiving user input 1910 m.

Turning now to FIG. 19T, virtual environment 1902 is configured as shownin FIG. 19M, with the addition of menu affordance 1924, which isassociated with objects in the vicinity of menu affordance 1924 (e.g.,table 1912, photos 1908, and mug 1918). In FIG. 19T, while gaze 1906 ispositioned on menu affordance 1924, device 1900 receives user input 1910n (e.g., a touch). In response to receiving user input 1910 n, menuaffordance 1924 is selected and gaze 1906 is disengaged from user input,as shown in FIG. 19U.

Selection of menu affordance 1924 causes display of menu options 1926a-1926 d, which can be cycled through and selected with inputindependent of gaze 1906. As shown in FIG. 19U, menu option 1926 a(SELECT TABLE) is initially designated with focus indicator 1928 (e.g.,bold border) in response to selection of menu affordance 1924.

As shown in FIG. 19V, device 1900 receives user input 1910 o, whichincludes a downward swipe or drag gesture. In response to receiving userinput 1910 o, focus indicator 1928 moves according to user input 1910 ofrom menu option 1926 a (SELECT TABLE) down to menu option 1926 b(SELECT PHOTOS), regardless of the position of gaze 1906, as shown inFIG. 19W.

In FIG. 19W, device 1900 receives user input 1910 p, which includesadditional downward movement. In response to receiving user input 1910p, focus indicator 1928 moves according to user input 1910 p from menuoption 1926 b (SELECT PHOTOS) down to menu option 1926 c (SELECT MUG),again regardless of the position of gaze 1906, as shown in FIG. 19X.

In FIG. 19X, while menu option 1926C is designated, device 1900 receivesuser input 1910 q. In response to receiving user input 1910 q, theobject corresponding to menu option 1926 c is selected, as shown in FIG.19Y. In FIG. 19Y, in response to receiving user input 1910 r, mug 1918is selected and moved to the center of the field of view of user 200.

Furthermore, although the embodiments described above with reference toFIGS. 19A-19Y refer to a virtual environment, similar techniques can beapplied to other CGR environments, including mixed reality environments.

Turning now to FIG. 20, a flow chart of exemplary process 2000 forinteracting with an electronic device using an eye gaze is depicted.Process 2000 can be performed using a user device (e.g., 100 a, 300,900, or 1900). The user device is, for example, a handheld mobiledevice, a head-mounted device, or a head-up device. In some embodiments,process 2000 is performed using two or more electronic devices, such asa user device that is communicatively coupled to another device, such asa base device. In these embodiments, the operations of process 2000 aredistributed in any manner between the user device and the other device.Further, the display of the user device can be transparent or opaque.Process 2000 can be applied to CGR environments, including virtualreality and mixed reality environments, and to virtual objects, physicalobjects, and representations (e.g., affordances) corresponding tovirtual and physical objects. Although the blocks of process 2000 aredepicted in a particular order in FIG. 20, these blocks can be performedin other orders. Further, one or more blocks of process 2000 can bepartially performed, optionally performed, combined with anotherblock(s), and/or additional blocks can be performed.

At block 2002, the device receives a first user input at a first time(e.g., a contact on a touch-sensitive surface, a press of a button, or abody gesture). At block 2004, in response to receiving the first userinput, the device designates a selection point at a first position basedon the gaze position at the first time, where the first positioncorresponds to the gaze position at the first time. In some embodiments,the gaze position at the first time is determined (e.g., measured orestimated) prior to the first user input (e.g., the most recent measuredor estimated position) or subsequent (e.g., in response) to the firstuser input.

In some embodiments, a focus indicator is displayed at the gazeposition. In some embodiments, the focus indicator includes a pointer,cursor, dot, sphere, highlighting, outline, or ghost image (e.g., of adesignated or selected object(s)). In some embodiments, the focusindicator designates a selection point corresponding to an object at thegaze position. In some embodiments, the device disengages gaze from userinput in response to the first user input.

In some embodiments, the response to the first input is contextual(e.g., the response is based on what is located at the gaze position(e.g., an object, a plurality of unresolvable objects, a menuaffordance, or no object) or whether an object is currently selected.For example, if an object is not currently selected, the device operatesin selection mode and performs a selection action (e.g., multipleobjects are displayed for selection), whereas if an object is currentlyselected, the device operates in placement mode and performs a placementaction (e.g., a ghost image of the selected object is displayed at atentative placement position).

In some embodiments, the selection point is designated at the firstposition in accordance with a determination that the first positioncorresponds to a plurality of selectable objects. In some embodiments, aplurality of objects is a group of closely spaced objects that cannot beresolved based on the gaze of the user. In some embodiments, theplurality of objects are menu options associated with an object (e.g., amenu affordance) at the gaze position. For example, the devicetentatively designates a selection point rather than confirmingselection of an object if it is determined that there is ambiguity oruncertainty about the object to be selected. In some such embodiments,in response to receiving the first user input, in accordance with adetermination that the first position corresponds to only a singleselectable object, the device selects the single selectable object(e.g., instead of designating the selection point at the firstposition).

In some embodiments, the selection point is designated at the firstposition in accordance with a determination that the first user input isa first type of input (e.g., a touch on a touch-sensitive surface, apress of a button, or a body gesture). In some such embodiments, inresponse to receiving the first user input and in accordance with adetermination that the first user input is a second type of inputdifferent than the first type of input (e.g., a click (in contrast to atouch) on the touch-sensitive surface, a press of a different button, ora different body gesture), the device confirms the selection point atthe first position.

While maintaining designation of the selection point, the deviceperforms the operations of blocks 2006, 2008, 2010, and 2012. At block2006, the device receives a second user input (e.g., movement of acontact on a touch-sensitive surface or press of a directional button).At block 2008, in response to receiving the second user input, thedevice moves the selection point to a second position different than thefirst position, where moving the selection point to the second positionis not based on the gaze position. For example, the device moves a focusindicator to a different object, selection point, or placement point. Insome embodiments, the selection point is moved based solely oncharacteristics (e.g., position, direction, speed, duration, etc.) ofthe second input. In some embodiments, the movement of the selectionpoint is independent from (not based on) a gaze position. In someembodiments, the second position is different than a gaze positionassociated with the second user input.

In some embodiments, the first user input is received while the deviceis in a first mode (e.g., a gaze-engaged mode in which a response to auser input is based on a gaze of a user), and the selection point isdesignated at the first position in accordance with the device being inthe first mode. In some such embodiments, in response to receiving thefirst user input, the device switches from the first mode to a secondmode (e.g., a gaze-disengaged mode in which a response to a user inputis not based on a gaze of a user such that the gaze is disengaged fromfurther input). In some such embodiments, the second input is receivedwhile the device is in the second mode, and the selection point is movedto the second position in accordance with the device being in the secondmode.

At block 2010, while the selection point is at the second position, thedevice receives a third user input (e.g., click, double tap, or contactliftoff from a touch-sensitive surface). At block 2012, in response toreceiving the third user input, the device confirms the selection pointat the second position. In some embodiments, the device re-engages inputwith gaze upon confirmation (e.g., switches from gaze-disengaged mode togaze-engaged mode). In some embodiments, the third input is receivedwhile the device is in the second mode (gaze-disengaged mode), and theselection point is confirmed at the second position in accordance withthe device being in the second mode.

In some embodiments, confirming the selection point selects an objectcorresponding to the position of the selection point (e.g., the secondposition). For example, in response to receiving the third user input,the device selects an object corresponding to the second position.

In some embodiments, confirming the selection point places an object atthe position of the selection point. For example, before receiving thefirst user input, the device selects an object at a third positiondifferent than the second position, and in response to receiving thethird user input, places the object at the second position. In someembodiments, before receiving the first user input, the device selectsan object at a third position different than the second position, wherethe selection point is designated at the first position in accordancewith a determination that the first position corresponds to a pluralityof selectable placement positions (e.g., the device does not confirm aplacement position if there is ambiguity or uncertainty about the placeto be selected). In some such embodiments, in response to receiving thefirst user input and in accordance with a determination that the firstposition corresponds to only a single selectable placement position, thedevice places the object at the single selectable placement position(e.g., the device bypasses tentatively designating a placement positionif there is only one option).

In some embodiments, the first user input is the same type of input asthe second user input or the third user input. In some embodiments, asingle tap or press of a button is used to designate a point orobject(s), and another single tap or press of the same button is used toconfirm a designated point or object(s). In some embodiments, devicedecides which action to take based on the mode (e.g., selection mode orplacement mode) in which the device is operating.

Turning now to FIG. 21, a flow chart of exemplary process 2100 forinteracting with an electronic device using an eye gaze is depicted.Process 2100 can be performed using a user device (e.g., 100 a, 300,900, or 1900). The user device is, for example, a handheld mobiledevice, a head-mounted device, or a head-up device. In some embodiments,process 2100 is performed using two or more electronic devices, such asa user device that is communicatively coupled to another device, such asa base device. In these embodiments, the operations of process 2100 aredistributed in any manner between the user device and the other device.Further, the display of the user device can be transparent or opaque.Process 2100 can be applied to CGR environments, including virtualreality and mixed reality environments, and to virtual objects, physicalobjects, and representations (e.g., affordances) corresponding tovirtual and physical objects. Although the blocks of process 2100 aredepicted in a particular order in FIG. 21, these blocks can be performedin other orders. Further, one or more blocks of process 2100 can bepartially performed, optionally performed, combined with anotherblock(s), and/or additional blocks can be performed.

At block 2102, the device receives a first user input at a first time.At block 2104, in response to receiving the first user input, the devicedesignates a first object of a plurality of objects based on a gazeposition (e.g., the position of the plurality of objects corresponds tothe gaze position at the first time). In some embodiments, a focusindicator designates the first object. In some embodiments, theplurality of objects are highlighted or enlarged or menu optionscorresponding to a menu affordance at the gaze position are displayed.

In some embodiments, the first object is designated in accordance with adetermination that the gaze position at the first time corresponds to agaze at more than one object (e.g., the plurality of objects). Forexample, the device does not confirm selection if there is ambiguity oruncertainty about the object to be selected. In some such embodiments,in response to receiving the first user input and in accordance with adetermination that the gaze position at the first time corresponds to agaze at only a single selectable object, the device selects the singleselectable object.

In some embodiments, the first user input is received while the deviceis in a first mode (e.g., a gaze-engaged mode in which a response to auser input is based on a gaze of a user) and the first object isdesignated in accordance with the electronic device being in the firstmode. In some such embodiments, in response to receiving the first userinput, the device switches from the first mode to a second mode (e.g.,switches to a gaze-disengaged mode in which a response to a user inputis not based on a gaze of a user such that the gaze is disengaged fromfurther input). In some such embodiments, the second input is receivedwhile the electronic device is in the second mode, and the second objectis designated in accordance with the electronic device being in thesecond mode.

In some embodiments, the first object is designated in accordance with adetermination that the first user input is a first type of input (e.g.,a touch on a touch-sensitive surface, a press of a button, or a bodygesture). In some such embodiments, in response to receiving the firstuser input and in accordance with a determination that the first userinput is a second type of input different than the first type of input(e.g., a click (in contrast to a touch) on the touch-sensitive surface,a press of a different button, or a different body gesture), the deviceselects the plurality of objects.

While maintaining designation of the first object, the device performsthe operations of blocks 2106 and 2108. At block 2106, the devicereceives a second user input. At block 2108, in response to receivingthe second user input, the device ceases designation of the first objectand designates a second object of the plurality of objects (e.g., movesa focus indicator to a different object), where designating the secondobject is not based on the gaze position. In some embodiments, thesecond object is selected based solely on characteristics of the seconduser input (e.g., position, direction, speed, duration, etc.). In someembodiments, the second user input is at a second time and a position ofthe second object at the second time is different than a gaze positionat the second time.

While maintaining designation of the second object, the device performsthe operations of blocks 2110 and 2112. At block 2110, the devicereceives a third user input. At block 2112, in response to receiving thethird user input, the device selects the second object.

In some embodiments, after selecting the second object, the devicereceives a fourth user input at a second time. In response to receivingthe fourth user input and in accordance with a determination that thefourth user input is a first type of input, the device places the secondobject at a gaze position at the second time. In response to receivingthe fourth user input and in accordance with a determination that thefourth user input is a second type of input different than the firsttype of input, the device designates a placement point corresponding tothe gaze position at the second time. In some such embodiments whilemaintaining designation of the placement position, the device receives afifth user input and, in response to receiving the fifth user input,places the second object at a current position of the placement point.

In some embodiments, the first user input is the same type of input asthe second user input or the third user input. In some embodiments, asingle tap or press of a button is used to designate the first object,and another single tap or press of the same button is used to select thesecond object.

Turning now to FIG. 22, a flow chart of exemplary process 2200 forinteracting with an electronic device using an eye gaze is depicted.Process 2200 can be performed using a user device (e.g., 100 a, 300,900, or 1900). The user device is, for example, a handheld mobiledevice, a head-mounted device, or a head-up device. In some embodiments,process 2200 is performed using two or more electronic devices, such asa user device that is communicatively coupled to another device, such asa base device. In these embodiments, the operations of process 2200 aredistributed in any manner between the user device and the other device.Further, the display of the user device can be transparent or opaque.Process 2200 can be applied to CGR environments, including virtualreality and mixed reality environments, and to virtual objects, physicalobjects, and representations (e.g., affordances) corresponding tovirtual and physical objects. Although the blocks of process 2200 aredepicted in a particular order in FIG. 22, these blocks can be performedin other orders. Further, one or more blocks of process 2200 can bepartially performed, optionally performed, combined with anotherblock(s), and/or additional blocks can be performed.

At block 2202, the device selects an object (e.g., as described inprocess 2100). While maintaining selection of the object, the deviceperforms the operations of blocks 2204, 2206, 2208, 2210, 2212, and2214. At block 2204, the device receives a first user input at a firsttime. At block 2206, in response to receiving the first user input, thedevice designates a placement point at a first position based on a gazeposition at the first time, where the first position corresponds to thegaze position at the first time.

In some embodiments, the placement point is designated at the firstposition in accordance with a determination that the first user input isa first type of input (e.g., a touch on a touch-sensitive surface, apress of a button, or a body gesture). In some such embodiments, inresponse to receiving the first user input and in accordance with adetermination that the first user input is a second type of inputdifferent than the first type of input (e.g., a click (in contrast to atouch) on the touch-sensitive surface, a press of a different button, ora different body gesture), the device places the selected object at thefirst position.

In some embodiments, the placement point is designated at the firstposition in accordance with a determination that the first positioncorresponds to a plurality of selectable placement positions (e.g., thedevice does not confirm the placement position when there is ambiguityor uncertainty about the position to be selected for placement). In somesuch embodiments, in response to receiving the first user input and inaccordance with a determination that the first position corresponds toonly a single selectable placement position, the device places theselected object at the single selectable placement position.

While maintaining designation of the placement point, the deviceperforms the operations of blocks 2208, 2210, 2212, and 2214. At block2208, the device receives a second user input. At block 2210, inresponse to receiving the second user input, the device moves theplacement point to a second position different than the first position,where moving the placement point to the second position is not based ongaze position (e.g., a gaze position at the time of the second userinput). In some embodiments, the second position is different than agaze position associated with the second user input.

In some embodiments, the first user input is received while theelectronic device is in a first mode (e.g., a gaze-engaged mode in whicha response to a user input is based on a gaze position) and theplacement point is designated at the first position in accordance withthe electronic device being in the first mode. In some such embodiments,in response to receiving the first user input, the device switches fromthe first mode to a second mode (e.g., switch to a gaze-disengaged modein which a response to a user input is not based on a gaze of a usersuch that the gaze is disengaged from further input in response to thefirst input). In some such embodiments, the second user input isreceived while the electronic device is in the second mode, and theplacement point is moved to the second position in accordance with theelectronic device being in the second mode.

At block 2212, the device receives a third user input. At block 2214, inresponse to receiving the third user input, the device places theselected object at the second position and, optionally, de-selects theobject. In some embodiments, the first user input is the same type ofinput as the second user input or the third user input. In someembodiments, a single tap or press of a button is used to designate theplacement point at the first position, and another single tap or pressof the same button is used to place the selected object at the secondposition.

Executable instructions for performing the features of processes 2000,2100, and/or 2200 described above are, optionally, included in atransitory or non-transitory computer-readable storage medium (e.g.,memory(ies) 106) or other computer program product configured forexecution by one or more processors (e.g., processor(s) 102). Someoperations in process 2000 are, optionally, included in process 2100and/or process 2200 (e.g., block 2004 and/or block 2008 are included inblock 2104 and/or block 2108, respectively), some operations in process2100 are, optionally, included in process 2000 and/or process 2200(e.g., block 2202 includes block 2112), and some operations in process2200 are, optionally, included in process 2000 and/or process 2100(e.g., block 2112 includes block 2202). Further, some operations inprocesses 2000, 2100, and/or 2200 (e.g., blocks 2004, 2008, 2104, 2108,2206, and/or 2210) are, optionally, included in processes 1600, 1700,and/or 1800, and some operations in processes 1600, 1700, and/or 1800(e.g., blocks 1604, 1606, 1704, 1706, 1804, and/or 1806) are,optionally, included in processes 2000, 2100, and/or 2200.

As described above, one aspect of the present technology involves theuse of data about a user's gaze. Gaze information, in the presenttechnology, can be used to the benefit of users. For example, a user'sgaze can be used to infer a user's focus on a particular part of acomputer generated reality environment and to allow the user to interactwith particular objects in that part of the field of view. It ispossible, however, that some users may consider gaze information to besensitive or to be of a personal nature.

Entities that collect, use, transfer, store, or otherwise affect gazeinformation detected by a CGR system should comply with well-establishedprivacy policies and/or privacy practices. In particular, such entitiesshould implement and consistently use privacy policies and practicesthat are generally recognized as meeting or exceeding industry orgovernmental requirements for maintaining personal information dataprivate and secure. Such policies should be easily accessible by users,and should be updated as the collection and/or use of data changes. Gazeinformation from users should be collected for legitimate and reasonableuses of the entity and not shared or sold outside of those legitimateuses. Further, such collection/sharing should occur after receiving theinformed consent of the users. Additionally, such entities shouldconsider taking any needed steps for safeguarding and securing access tosuch information data and ensuring that others (if any) with access tothe gaze information data adhere to their privacy policies andprocedures. Further, such entities can subject themselves to evaluationby third parties to certify their adherence to widely accepted privacypolicies and practices. In addition, policies and practices should beadapted for the particular types of gaze information data beingcollected and/or accessed and adapted to applicable laws and standards,including jurisdiction-specific considerations.

The present disclosure also contemplates embodiments in which usersselectively block the use of, or access to, gaze information. Entitiesimplementing the present technology can determine whether certainfeatures can be provided while allowing users to select to “opt in” or“opt out” of participation in the use of gaze information duringregistration for services or anytime thereafter. In addition toproviding “opt in” and “opt out” options, the present disclosurecontemplates providing notifications relating to the access or use ofgaze information. For instance, a user may be notified upon downloadingan app that their personal gaze data will be accessed. Users may also betransparently educated as to why some gaze information is being used toprovide certain features. For example, in virtual reality systems wheregaze information is used to determine where a user is looking, user maybe informed that the their gaze information is being used to determinewhat field of view of the virtual environment the system should render,thereby enabling the user to make an informed decision as to when toallow the use of gaze information.

Nonetheless, it is the intent of the present disclosure that gazeinformation data should be managed and handled in a way to minimizerisks of unintentional or unauthorized access or use. Risk can beminimized by limiting the collection of data and deleting data once itis no longer needed. In addition, and when applicable, datade-identification can be used to protect a user's privacy.De-identification may be facilitated, when appropriate, by removingspecific identifiers (e.g., user name, device name, etc.), controllingthe amount or specificity of data stored (e.g., collect mathematicalcoordinates of where a user is looking in a coordinate system but avoidcollecting information on what content is being viewed at thecoordinates), controlling how data is stored (e.g., locally), and/orother methods.

The foregoing descriptions of specific embodiments have been presentedfor purposes of illustration and description. They are not intended tobe exhaustive or to limit the scope of the claims to the precise formsdisclosed, and it should be understood that many modifications andvariations are possible in light of the above teaching.

What is claimed is:
 1. An electronic device, comprising: one or moreprocessors; and memory storing one or more programs configured to beexecuted by the one or more processors, the one or more programsincluding instructions for: receiving a first user input at a firsttime; in response to receiving the first user input, designating a firstobject of a plurality of selectable objects, wherein a gaze position atthe first time corresponds to a gaze at the plurality of selectableobjects; while maintaining designation of the first object, receiving asecond user input; in response to receiving the second user input:ceasing designation of the first object; and designating a second objectof the plurality of selectable objects, wherein designating the secondobject is not based on the gaze position; while maintaining designationof the second object, receiving a third user input; in response toreceiving the third user input, selecting the second object; afterselecting the second object, receiving a fourth user input at a secondtime; and in response to receiving the fourth user input, in accordancewith a determination that the fourth user input is a first type ofinput, placing the second object at a gaze position at the second time.2. The electronic device of claim 1, the one or more programs furtherincluding instructions for: in response to receiving the first userinput, in accordance with a determination that the gaze position at thefirst time corresponds to a gaze at only a single selectable object,selecting the single selectable object.
 3. The electronic device ofclaim 1, the one or more programs further including instructions for: inresponse to receiving the fourth user input, in accordance with adetermination that the fourth user input is a second type of inputdifferent than the first type of input, designating a placement pointcorresponding to the gaze position at the second time.
 4. The electronicdevice of claim 3, the one or more programs further includinginstructions for: while maintaining designation of the placement point,receiving a fifth user input; and in response to receiving the fifthuser input, placing the second object at a current position of theplacement point.
 5. The electronic device of claim 1, wherein the firstuser input is received while the electronic device is in a first mode,and wherein the first object is designated in accordance with theelectronic device being in the first mode, the one or more programsfurther including instructions for: in response to receiving the firstuser input, switching the electronic device from the first mode to asecond mode, wherein the second user input is received while theelectronic device is in the second mode, and wherein the second objectis designated in accordance with the electronic device being in thesecond mode.
 6. The electronic device of claim 1, wherein the first userinput is the same type of input as the second user input or the thirduser input.
 7. The electronic device of claim 1, wherein the second userinput is at a second time, wherein a position of the second object atthe second time is different than a gaze position at the second time. 8.The electronic device of claim 1, wherein the first object is designatedin accordance with a determination that the first user input is a firsttype of input, the one or more programs further including instructionsfor: in response to receiving the first user input, in accordance with adetermination that the first user input is a second type of inputdifferent than the first type of input, selecting the plurality ofselectable objects.
 9. A non-transitory computer-readable storage mediumstoring one or more programs configured to be executed by one or moreprocessors of an electronic device, the one or more programs includinginstructions for: receiving a first user input at a first time; inresponse to receiving the first user input, designating a first objectof a plurality of selectable objects without designating other objectsof the plurality of selectable objects, wherein a gaze position at thefirst time corresponds to a gaze at the plurality of selectable objects;while maintaining designation of the first object receiving a seconduser input; in response to receiving the second user input: ceasingdesignation of the first object; and designating a second object of theplurality of selectable objects, wherein designating the second objectis not based on the gaze position; while maintaining designation of thesecond object, receiving a third user input; in response to receivingthe third user input, selecting the second object; after selecting thesecond object, receiving a fourth user input at a second time; and inresponse to receiving the fourth user input, in accordance with adetermination that the fourth user input is a first type of input,placing the second object at a gaze position at the second time.
 10. Thenon-transitory computer-readable storage medium of claim 9, the one ormore programs further including instructions for: in response toreceiving the first user input, in accordance with a determination thatthe gaze position at the first time corresponds to a gaze at only asingle selectable object, selecting the single selectable object. 11.The non-transitory computer-readable storage medium of claim 9, the oneor more programs further including instructions for: in response toreceiving the fourth user input, in accordance with a determination thatthe fourth user input is a second type of input different than the firsttype of input, designating a placement point corresponding to the gazeposition at the second time.
 12. The non-transitory computer-readablestorage medium of claim 11, the one or more programs further includinginstructions for: while maintaining designation of the placement point,receiving a fifth user input; and in response to receiving the fifthuser input, placing the second object at a current position of theplacement point.
 13. The non-transitory computer-readable storage mediumof claim 9, wherein the first user input is received while theelectronic device is in a first mode, and wherein the first object isdesignated in accordance with the electronic device being in the firstmode, the one or more programs further including instructions for: inresponse to receiving the first user input, switching the electronicdevice from the first mode to a second mode, wherein the second userinput is received while the electronic device is in the second mode, andwherein the second object is designated in accordance with theelectronic device being in the second mode.
 14. The non-transitorycomputer-readable storage medium of claim 9, wherein the first userinput is the same type of input as the second user input or the thirduser input.
 15. The non-transitory computer-readable storage medium ofclaim 9, wherein the second user input is at a second time, wherein aposition of the second object at the second time is different than agaze position at the second time.
 16. The non-transitorycomputer-readable storage medium of claim 9, wherein the first object isdesignated in accordance with a determination that the first user inputis a first type of input, the one or more programs further includinginstructions for: in response to receiving the first user input, inaccordance with a determination that the first user input is a secondtype of input different than the first type of input, selecting theplurality of selectable objects.
 17. A method comprising: at anelectronic device that is in communication with one or more inputdevices: receiving, via the one or more input devices, a first userinput at a first time; in response to receiving the first user input,designating a first object of a plurality of selectable objects withoutdesignating other objects of the plurality of selectable objects,wherein a gaze position at the first time corresponds to a gaze at theplurality of selectable objects; while maintaining designation of thefirst object, receiving, via the one or more input devices, a seconduser input; in response to receiving the second user input: ceasingdesignation of the first object; and designating a second object of theplurality of selectable objects, wherein designating the second objectis not based on the gaze position; while maintaining designation of thesecond object, receiving, via the one or more input devices, a thirduser input; in response to receiving the third user input, selecting thesecond object; after selecting the second object, receiving, via the oneor more input devices, a fourth user input at a second time; and inresponse to receiving the fourth user input, in accordance with adetermination that the fourth user input is a first type of input,placing the second object at a gaze position at the second time.
 18. Themethod of claim 17, the method further comprising: in response toreceiving the first user input, in accordance with a determination thatthe gaze position at the first time corresponds to a gaze at only asingle selectable object, selecting the single selectable object. 19.The method of claim 17, further comprising: in response to receiving thefourth user input, in accordance with a determination that the fourthuser input is a second type of input different than the first type ofinput, designating a placement point corresponding to the gaze positionat the second time.
 20. The method of claim 19, further comprising:while maintaining designation of the placement point, receiving, via theone or more input devices, a fifth user input; and in response toreceiving the fifth user input, placing the second object at a currentposition of the placement point.
 21. The electronic device of claim 1,wherein the first object of the plurality of selectable objects isdesignated without designating other objects of the plurality ofselectable objects.
 22. The non-transitory computer-readable storagemedium of claim 9, wherein the first object of the plurality ofselectable objects is designated without designating other objects ofthe plurality of selectable objects.
 23. The method of claim 17, whereinthe first object of the plurality of selectable objects is designatedwithout designating other objects of the plurality of selectableobjects.